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Abstract 

Each  year  the  Air  Force  awards  weapon  system  contracts 
worth  billions  of  dollars  using  the  source  selection 
process.  In  the  source  selection,  performance  cuid  risk 
assessments  are  evaluated  to  determine  the  best  proposal ; 
the  proposal  that  gives  the  best  overall  value. 

The  purpose  of  this  research  was  to  investigate  the 
performance/risk  trade-off  in  the  source  selection  decision 
making  process.  The  study  also  examined  the  effect 
individual  risk  propensities  cUid  the  nature  of  item  being 
evaluated  have  on  the  performance/risk  trade-off  decisions. 

Air  Force  acquisition  personnel  stationed  at 
Aeronautical  Systems  Center,  Wright -Patterson  AFB,  OH,  who 
had  participated  in  a  source  selection  were  surveyed. 
Respondents  were  asked  to  rank  the  color -risk  combinations 
used  in  a  source  selection  from  most  to  least  preferred. 
Respondents  indicated  their  degree  of  preference  between 
color-risk  rating  pairs.  Ccxi^arlsons  were  made  for  two  item 
summary  levels:  technical  capability  euid  mcuiagement. 

A  definite  rank  order  exists  among  color-risk  ratings. 
The  rank  is  as  follows:  Blue-Low,  Green- Low/Blue -Moderate 
(tied),  Green-Moderate,  Blue-High,  Yellow-Low,  Green-High, 
Yellow-MOderate,  and  Yellow-High.  Risk  propensity  cUid  item 
stimmary  level  do  not  significantly  influence  the  reuik  order. 


AN  ANALYSIS  OF  THE  PERFORMANCE -RISK 
TRADE-OFF  IN  SOURCE  SELECTION  DECISION  MAKING 


I .  introduction 


Gengygil  Issue 

Each  year  the  Air  Force  (AF)  awards  many  weapon  system 
contracts  worth  billions  of  dollars.  At  completion,  some  of 
these  contracts  exceed  budget  and  schedule  performance 
requirements  estcdDlished  at  contract  award.  These  overruns 
are  of  great  concern  to  AF  programming  and  planning 
personnel,  especially  at  the  higher  echelons,  in  today's 
acquisition  environment  where  budgets  are  tight,  scrutiny 
comes  from  many  sources,  both  internal  and  external  to  the 
program.  The  primary  concern  is  that  capable  and  cost 
effective  weapon  systems  are  purchased  within  baselined 
standards . 

Source  selection  is  the  process  the  AF  uses  to 
competitively  award  weapon  system  contracts.  The 
regulations  covering  the  source  selection  process  state  the 
principal  objective  is: 

to  select  the  offeror  whose  proposal  has  the 
highest  degree  of  credibility,  and  whose 
performance  can  best  meet  the  government ' s 
requirements  at  an  affordcdsle  cost... The  process 
must  be  efficient  and  capable  of  balancing 


technical,  cost,  and  business  considerations 
consistent  with  requirements. . . (5:3) . 

These  subjectively  evaluated  standards  are  intended  to 

provide  a  framework  for  arriving  at  award  decisions  without 

constraining  any  options  availcdsle  to  the  source  selection 

team.  There  are  no  mandates  to  award  a  contract  based  on 

lowest  bid,  shortest  schedule,  highest  performance,  or 

lowest  risk.  These  elements  are  weighed  along  with  business 

considerations  to  determine  the  most  advauitageous  approach 

to  meet  the  government ' s  needs . 

Of  all  the  elements  stated  above,  risk  is  the  area  that 

presents  the  greatest  management  challenge  to  the  source 

selection  team.  The  regulations  give  the  source  selection 

team  the  most  flexibility  in  dealing  with  risk.  The 

regulations  require  the  source  selection  team  to  establish  a 

strategy  to  minimize  risk,  but  the  regulations  do  not  define 

risk.  The  definition  of  risk  is  left  to  the  individuals 

involved  in  the  source  selection.  These  individuals  make 

decisions  concerning  risk  based  upon  their  preferences  and 

experiences  (15:22) .  The  results  of  source  selection 

decisions  can  vary  due  to  individual  perceptions  of  risk. 

Therefore,  understanding  one's  perspective  on  risk  is 

critical  when  evaluating  the  reasons  for  a  specific 

decision. 
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In  order  to  determine  the  role  of  risk  in  management 
decisions,  MacCrimmon  and  wehrung  assert  that  "the  first 
priority  should  be  to  study  people  whose  profession  it  is  to 
make  risky  decisions.  Prime  candidates  then,  are  managers 
at  the  top  levels  of  business  firms"  (15:53).  In  the  AF, 
these  managers  are  acquisition  professionals  participating 
in  the  source  selection  process. 

Problem  Statement 

In  1969,  Deputy  Secretary  of  Defense  David  Packard 
wrote  a  memorandiam  to  the  services  listing  inadequate  risk 
assessment  as  a  major  problem  area  in  system  acquisition 
(3:2-1) .  In  1981,  Deputy  Secretary  of  Defense  Frank  C. 
Carlucci,  III,  published  a  memorandum  aimed  at  improving  the 
acquisition  process.  It  required  the  Department  of  Defense 
action  to  increase  the  visibility  of  technical  risk  of 
weapon  systems  acquisition  programs  (3:2-1).  Risk 
assessment  and  management  are  important  elements  in  system 
acquisitions  to  Department  of  Defense  leadership. 

In  the  source  selection  environment,  cost,  schedule  and 
technical  performance  risk  are  evaluated  and  traded- off  to 
arrive  at  an  award  decision  (5:3).  "While  cost  and  schedule 
are  understood,  the  impact  of  cost/schedule  decisions  as 
they  relate  to  technical  performance  risk  are  usually  not  as 
clear"  (3:2-2, 2-3).  Since  the  effects  of  technical 
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performance  risk  are  not  well  understood,  this  area  needs 
further  study. 

The  problem  being  addressed  by  this  study  is  the 
effects  of  performance/risk  trade-off  decisions  in  the 
source  selection  environment. 

Research  Objective 

The  purpose  of  this  research  is  to  determine  whether 
participants  in  the  source  selection  process  are  consistent 
in  their  performance /risk  trade-off  decisions  or  whether 
there  are  differences  based  on  individual  risk  propensity. 
It  also  assesses  whether  such  trade-offs  vary  by  the  nature 
of  the  item  being  evaluated. 

Investigative  Questions 

To  meet  the  objective  of  our  research  question,  the 
following  five  investigative  questions  need  to  be  answered: 

1.  Vfliat  is  the  risk  propensity  of  the  source 
selection  participants? 

2.  How  do  source  selection  participants  trade-off 
performance  versus  risk? 

3.  Does  risk  propensity  influence  the  trade-off 
decision? 

4.  Does  the  performance  versus  risk  trade-off 
decision  vary  based  on  the  item  being 
evaluated? 
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5.  To  what  degree  are  color-risk  ratings 
preferred  over  one  another? 

Assumptions  and  Limitations 

Several  assumptions  were  made  which  limit  the  scope  of 
this  investigation  and  the  applicability  of  this  research. 

The  first  assvunption  is  that  the  response  of 
individuals  surveyed  accurately  reflects  how  they  would 
decide  when  participating  in  an  actual  source  selection. 

The  questions  provided  in  our  survey  are  hypothetical .  As 
such,  the  survey  respondents  are  presented  with  imaginary 
scenarios.  MacCrimmon  and  Wehrung  report  that  when  asked  to 
respond  in  this  manner,  people  tend  to  view  risk  as  less 
threatening  (15:121).  This  tendency  can  cause  individuals 
to  appear  more  risk  prone  than  they  actually  are. 

The  next  assun?)tion  is  that  survey  respondents  will 
answer  the  survey  with  respect  to  their  professional 
propensity  for  dealing  with  risk.  MacCrimmon  and  Wehrung 
found  that  individuals  treat  the  trade-off  of  money  versus 
risk  differently  in  their  personal  and  professional  lives 
(15:176).  This  research  attenpts  to  minimize  this  effect  by 
distributing  the  survey  in  the  work  place  cuid  asking 
participants  to  make  trade-off  decisions  in  a  government 
source  selection  context. 

The  final  assumption  is  that  individuals  who  have 
participated  in  source  selections  are  not  biased  by  whether 
the  source  selection  was  formal  or  streamlined. 
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One  limitation  of  this  research  effort  stems  from 
differences  in  individual  perceptions  concerning  relative 
amounts  of  money.  Variances  in  individual  responses  can 
occur  because  of  the  different  sizes  of  programs. 

Incremental  changes  in  amounts  of  money  are  viewed  with 
relativity  (24:454).  "A  $1,000  change  of  costs  frcan  $1,000 
to  $2,000  is  typically  more  significant  than  from  $60,000  to 
$61,000.  The  choice  of  dollar  figures  could  influence  risk 
assessment"  (22:12).  Another  example  of  this  is  that  the 
relative  value  of  $100,000  to  a  person  who  typically  deals 
with  a  million  dollar  contracts  is  different  from  one  who 
deals  with  billion  dollar  contracts. 

There  are  limitations  associated  with  the  research 
sample.  The  sample  was  taken  from  people  who  have 
participated  in  source  selections  at  Wright -Patterson  Air 
Force  Base  (WPAFB)  Aeronautical  Systems  Center  (ASC)  and  are 
still  assigned  there.  This  sample  population  will  not  be  a 
perfect  indicator  of  future  source  selection  behavior  as  it 
does  not  contain  anyone  who  will  participate  on  a  ASC  source 
selection  team  for  the  first  time.  A  typical  source 
selection  team  consists  of  a  core  of  experienced  people 
(those  who  have  participated  in  prior  source  selection)  and 
some  people  who  are  participating  for  the  first  time. 

The  research  is  further  limited  by  the  fact  that  it 
cannot  completely  address  all  the  trade-offs  involved  in  the 
final  award  decision.  The  best  indicator  of  how  source 
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selection  decisions  are  made  resides  with  the  Source 
Selection  Authority  (SSA) ,  as  the  SSA  maUces  the  final  award 
decision.  The  population  of  SSAs  is  very  small  at  ASC  as 
the  dollar  thresholds  of  the  weapon  systems  being 
acquired  typically  require  the  source  selection  authorities 
to  be  at  least  the  Commander  of  ASC.  Since  only  two  or 
three  SSAs  would  be  available  for  study,  no  significant 
statistical  conclusions  could  be  drawn  from  data  collected 
on  SSAs.  Although  the  research  does  not  profile  the  actual 
decision  maker,  it  does  measure  the  decisions  of  those  who 
have  the  potential  to  influence  SSAs  the  most.  The  source 
selection  evaluators  also  possess  the  potential  to  become 
SSAs. 

Overview  of  the  Thesis  Structure 

This  chapter  has  presented  the  general  issue,  problem 
statement,  research  objectives,  investigative  cjuestions,  and 
assumptions  and  limitations  of  this  research  effort. 

Chapter  II  examines  general  risk  concepts,  risk  related 
behavior,  and  source  selection  risk  management  guidance  in 
the  Department  of  Defense  (DoD) .  Based  on  the  current 
literature,  it  supports  the  research  of  source  selection 
team  members'  risk  propensity  based  on  the  use  of  a  source 
selection  evaluation  matrix. 

Chapter  III  discusses  the  methodology  used  to  guide  the 
research  effort  toward  answering  the  investigative  questions 
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put  forth  earlier  in  this  chapter.  Chapter  III  also 
discusses  the  method  of  data  collection  emd  the  methods  of 
analysis. 

Chapter  IV  presents  the  results  of  the  study. 
Statistical  tests  euid  levels  of  significance  are  used  to 
relate  findings  back  to  the  investigative  questions  posed. 

Finally,  chapter  V  contains  conclusions  and  suggestions 
for  areas  of  further  research. 


II.  Literature  Review 


Introduction 

Nowhere  is  risk  infoinnation  being  used  more  in  the 
Department  of  Defense  (DoD)  than  in  systems  acquisition 
(22:36-37)  .  MacCrimmon  and  Wehrting  assert  in  their  book 
Taking  Risks  that: 

the  first  priority  [when  studying  risk]  should  be 
to  study  people  whose  profession  it  is  to  maUce 
risky  decisions.  Prime  camdidates  then,  are 
managers  at  the  top  levels  of  business  firms 
(15:53)  . 

Therefore,  in  order  to  effectively  study  risk  management  in 
DoD  systems  acquisition,*  these  decision  makers  need  to  be 
studied.  This  literature  review  examines  general  risk 
concepts,  risk  related  behavior,  and  DoD's  source  selection 
risk  management  guidance.  The  review  supports  the  study  of 
the  performance/risk  trade-off  in  source  selection  decision 
making . 

The  literature  review  begins  with  an  examination  of 
various  definitions  of  risk  culminating  in  the  establishment 
of  a  working  definition  for  this  review.  The  ideas  of  the 
presence  of  risk,  the  need  for  risk  and  people's  behavior 
when  confronting  risk  are  developed.  Next,  the  various 
methods  of  studying  individual  risk  propensity  are 
discussed.  This  examination  provides  the  framework  in  which 
to  investigate  DoD  and  Air  Force  (AF)  risk  manag^ent 
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policies  as  they  relate  to  general  management  practices  and 
the  source  selection  process,  respectively.  Previous 
research  on  risk  in  the  source  selection  process  will  be 
examined.  The  examination  of  previous  research  will 
culminate  in  a  statement  of  this  research  topic ' s 
significance. 


RisH 


In  order  to  effectively  research  risk,  a  clear 
definition  of  risk  must  be  given.  The  following  five 
definitions  are  representative  of  the  many  sources 
consulted: 

1.  Exposure  to  the  chance  of  injury  or  loss 
{19:1139) . 

2.  First,  it  is  necessary  that  there  be  a 
potential  loss  of  some  amount.  Second,  there 
must  be  a  chance  of  loss.  A  sure  loss  is  not 
a  risk.  Third,  the  notion  "to  expose"  means 
that  the  decision  maker  can  take  actions  that 
can  increase  the  magnitude  or  chance  of  loss 
(15:9)  . 

3.  The  probability  of  an  undesirable  event 
occurring  amd  the  significance  of  the 
consequence  of  the  occurrence  (3:3-1). 

4.  A  totally  inclusive  definition  of  risk 
includes,  dangers,  potential  for  loss,  the 
degree  or  probcd}ility  of  a  specific  exposure 
for  loss  as  well  as  the  liability  to  injury, 
damage,  loss,  or  pain.  It  enconqpasses 
jeopardy  or  the  exposure  to  extreme  danger  for 
any  situation.  Events  with  both  chauice  and 
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voluntary  provocation  are  included.  Loss 
potential  due  to  risk  also  embraces  rational 
behavior,  irrational  behavior,  natural 
phenomena,  and  any  other  potential  for 
realizing  unwanted,  negative  consequences  of 
any  event  (9:24) . 

5.  The  condition  of  having  outcomes  with  known 
probabilities  of  occurrence,  not  certainty  of 
occurrence  ( 2 : B - 5 ) . 

A  common  element  in  each  of  these  definitions  is  the 
potential  for  some  type  of  loss.  This  raises  another 
question  that  must  be  answered;  what  is  a  loss?  Definitions 
of  loss  include: 

1.  The  state  of  being  deprived  of  or  of  being 
without  something  that  one  has  had  (19:792). 

2.  An  outcome  that  will  make  us  worse  off  than 
some  reference  status  quo  position  (15:10). 

3.  An  outcome  that  is  not  as  good  as  some  other 
outcome  that  might  have  been  obtained  (15:10) . 

The  concept  of  loss  helps  establish  a  good  working 
definition  for  the  purpose  of  this  study:  risk  involves 
some  type  of  loss  (undesirable  event)  coupled  with  some  type 
of  chance.  Now  that  a  definition  of  risk  has  been 
established,  investigation  of  risk  itself  can  begin. 


Why  Do  We  Have  Risks? 

Risks  are  present  in  every  daily  activity.  Risk  arises 
in  scmie  form  in  virtually  all  fields  of  endeavor.  It  is 
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in^ortant  neither  to  ignore  risk,  nor  to  be  frightened  by  it 
(17:ix) .  This  is  why  it  is  so  important  to  develop  an 
effective  way  to  deal  with  risk.  If  managers  can  be  certain 
of  nothing  else,  they  can  be  certain  that  they  will  be  faced 
with  choices,  that  in  some  degree,  involve  risk. 

Life  requires  choices;  choices  require  risks. 

While  you  can  choose  to  minimize  the  risks  you 

face,  you  cannot  avoid  risks  completely.  Along 

with  death  and  taxes,  risk  is  one  of  the 

certainties  of  life  (15:4). 

"Our  choices  do  not  just  affect  us,  they  affect  many 
other  people.  Business  and  government  decisions  can 
influence  the  risks  faced  by  thousands  or  millions  of 
others"  (15:3).  Managers,  executives,  and  all  kinds  of 
leaders  make  decisions  involving  risks.  The  decisions  they 
make  not  only  affect  their  own  lives  and  futures,  but  also 
the  lives  of  others  to  whom  their  decisions  are  linked. 

Knowing  that  choices  involving  risks  are  always  present 
and  have  tremendous  consequences  brings  forth  two  questions: 

1.  Why  do  decision  makers  take  risks? 

2.  What  motivates,  drives,  and  possesses  decision 
makers  to  make  risky  decisions,  Jcnowing  that  their 
choice  can  impact  so  many  others? 

In  regard  to  these  questions,  March  and  Shapira's  research 

concludes : 

1.  Risk  is  essential  to  success  in  decision 
making . 

2 .  Risk  taking  is  an  essential  component  of  the 
managerial  role. 
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3 .  Managers  recognize  emotional  pleasures  and 
pains  of  risk  taking,  the  affective  delights 
and  thrills  of  danger.  Risk  taking  involves 
emotions  of  anxiety,  fear,  stimulation,  and 
joy  (16:1409) . 

The  risks  that  decision  makers  encounter  come  from  a 
variety  of  sources.  MacCrimmon  and  Wehrung  state,  as  a 
result  of  their  research,  that  there  are  three  basic 
identifiable  determinants  of  risk:  lack  of  control,  lack  of 
infoannation,  and  lack  of  time.  Risk  does  not  exist  if  the 
decision  maker  has  complete  control  over  the  situation  and 
can  determine  the  optimal  outcome.  Also,  if  the  decision 
maker  has  perfect  knowledge  of  the  situation,  risk  is 
eliminated  because  the  decision  maker  can  choose  the  best 
alternative.  Finally,  if  the  decision  maker  is  not  limited 
by  time,  the  choice  can  be  delayed  until  the  risk  is 
eliminated  (15:14-15). 

MacCrimmon  and  Wehrung  also  describe  three  components 
of  risk:  magnitude  of  potential  loss,  chances  of  potential 
loss,  and  exposure  to  potential  loss.  To  reduce  risk  in  a 
situation,  one  of  these  three  components  must  be  reduced. 

If  the  decision  maker  can  reduce  all  three  of  the 
components,  the  situation  might  not  involve  any  risk  at  all 
(15:10).  The  outcome  of  risk  is  directly  related  to  the 
three  determinants  and  three  ccxnponents  of  risk.  Table  i 
illustrates  the  impact  on  the  decision  maker  under  the 
various  combinations  of  determinants  and  components  of  risk. 
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All  Of  the  risk  determinants  and  conqponents  need  to  be 


addressed  by  not  only  managers  and  executives,  but  by 
everyone  in  day-to-day  activities.  The  way  a  manager  deals 


Tad>le  1 

Relationships  Between  Components  (columns)  and 
Determinants  of  Risk  (rows) (15 : 19) 


Magnitude  of 
Potential 

Loss 

Chances  of 
Potential 

Loss 

Exposure  to 
Potential 

Loss 

Lack  of 
Control 

Cannot 
affect  size 
of  potential 
loss 

Cannot 

affect 

chances  of 

potential 

loss 

Cannot 

affect 
exposure  to 
potential 
loss 

Lack  of 
Information 

Do  not  know 
size  of 
potential 
loss 

Do  not  know 
chances  of 
potential 
loss 

Do  not  know 
exposure  to 
potential 
loss 

Lack  of  Time 

Insufficient 
time  to 
understand 
or  reduce 
magnitude  of 
potential 
loss 

Insufficient 
time  to 
understand 
or  reduce 

chances  of 

potential 

loss 

Insufficient 
time  to 
understand 

or  reduce 
exposure  to 
potential 
loss 

with  these  determinants  and  components  of  risk  varies  from 
individual  to  individual.  These  variances  lead  to  an  area 
of  risk  that  must  be  analyzed;  individual  behavior  of  those 
who  confront,  handle,  and  make  choices  concerning  risk. 
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Risk  Behavior 

Everyone,  regardless  of  their  position,  goes  through  a 

process  to  manage  the  determinants  and  components  of  risk. 

How  a  manager  chooses  to  accomplish  this  process,  is 

directly  related  to  the  manager's  risk  propensity.  The 

process  varies  from  individual  to  individual,  but  the 

process  takes  a  common  form;  some  type  of  choice  involved, 

where  one  of  the  choices  includes  a  chance  of  a  loss. 

This  focus  on  choice  captures  the  core  of  a  risky 
situation  -  but  only  the  core.  At  some  point  a 
person  will  make  a  choice,  but  a  lot  of  things  can 
happen  before  the  choice  is  made.  Knowing  these 
things  can  often  tell  us  more  about  risk  behavior 
than  the  choice  itself  (15:20-21) . 

Before  managers  can  choose  among  alternative  options, 

they  must  first  recognize  that  a  risky  situation  exists. 

A  person  acquires  some  information  about  a 
situation  that  may  result  in  his  losing  something 
of  value.  Although  an  omniscient  outside  observer 
may  be  able  to  make  an  "objective"  appraisal  of 
the  risks,  the  decision  maker  himself  may  perceive 
the  situation  in  a  very  different  way 
(15:21-22) . 

The  decision  maker's  choice  is  dependent  upon  how  he 

perceives  the  risks  involved  with  his  alternatives.  These 

same  perceptions  then  become  part  of  the  second  step, 

evaluating  the  risks  in  a  particular  situation. 

The  individual  overtly  acts  to  expose  himself  to 
some  risky  situations;  others  he  overtly  acts  to 
avoid,  while  others  he  accepts  as  they  confront 
him.  In  each  case  the  individual  evaluates  the 
acceptability  of  the  risks  (15:23-24) . 
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After  the  decision  maker  recognizes  and  evaluates  the 
risk  in  a  particular  situation,  adjustments  can  be  made. 

How  or  even  if  the  decision  maker  decides  to  adjust  a  risk, 
depends  upon  how  active  or  passive  the  decision  maker  is. 
Individuals  that  display  active  behavior  will  try  to  adjust 
the  risks,  while  those  who  display  passive  behavior  will 
simply  choose  among  the  existing  alternatives  (15:24-25). 

If  a  decision  maker  decides  that  the  risks  need  to  be 
adjusted,  there  are  three  basic  ways  to  adjust  the  risks. 

The  decision  maker  can  try  to  gain  more  control  over  the 
situation,  more  information  about  the  situation,  or  more 
time  to  deal  with  the  situation  (15:28) . 

After  all  of  this  has  been  accomplished,  the  decision 
maker  arrives  at  the  crux  of  managing  risk:  choosing  among 
the  alternatives.  The  decision  maker  must  select  an 
alternative  based  on  what  cannot  be  adjusted.  This  choice 
is  directly  related  to  the  risk  propensity  of  the  decision 
maker  (15:25) . 

When  it  comes  to  risk  propensity,  people  can  be 
categorized  into  one  of  three  categories:  risk  averter,  risk 
neutral,  and  risk  taker  (15:34) .  Risk  averters,  risk 
neutrals,  and  risk  takers  act  differently  when  managing 
risks. 

A  risk  taker  would  accept  a  higher  exposure  in  the 
sense  of  taking  sole  responsibility,  acting  with 
less  information,  and  requiring  less  control  than 
would  a  risk  averter.  The  risk  taker  would  accept 
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a  higher  chance  of  loss,  would  operate  in 
unfamiliar  situations,  would  tolerate  more 
uncertainty,  and  would  require  less  information 
about  the  chances  [than  would  a  risk  averter] 

(15:34) . 

Risk  neutrals  would  fall  somewhere  between  risk  takers  and 
risk  averters;  their  actions  displaying  some  tendencies  of 
both  characteristics  (15:93). 

The  categorization  of  an  individual's  risk  propensity 
is  partially  dependent  upon  several  uncontrollable  factors. 
The  following  researchers  have  studied  some  of  these 
factors.  Hudgens  and  Fatkin  found  that  men  were  observed  to 
be  more  likely  to  make  decisions  involving  greater  risks 
than  women  (12:204) .  Hayden  and  Thomas  found  that  military 
rank,  line/staff  designation  ^nd  sex  influence  risk 
propensity  (11:64-76).  For  example,  male,  high  ranking, 
line  officers  were  more  risk  seeking  than  lower  ranking, 
female,  staff  officers.  Hamilton's  research  demonstrated 
Atkinson's  theory:  people  who  have  high  need  for  success 
tend  to  be  more  risk  seeking  than  those  who  have  a  high  need 
to  avoid  failure  (lO) . 

MacCrimmon  and  Wehrung  conducted  a  comprehensive  study 
of  these  uncontrol laible  factors  on  over  500  top  level 
American  and  Canadian  managers  (I5:xii) .  They  investigated 
the  relationship  between  risk  propensity  and  personal 
factors  such  as  age,  education,  nationality,  sex,  or 
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management  level .  They  also  studied  the  relationship 
between  risk  propensity  and  situational  factors  such  as  firm 
size,  industry,  and  whether  the  decision  was  business  or 
personal  in  nature  (15:xii) .  Their  findings  include: 

1.  Younger  managers  take  more  risks  than  older 
managers , 

2.  Managers  with  few  dependents  take  more  risks  than 
managers  with  many  dependents, 

3 .  Managers  with  postgraduate  education  take  more 
risks  than  managers  with  a  bachelor's  degree  or 
less, 

4.  Managers  with  more  wealth  take  more  risks, 

5.  Managers  with  higher  incomes  take  more  risks, 

6.  Higher  level  managers  take  more  risks  than  lower 
level  ones, 

7.  Managers  with  more  authority  take  more  risks, 

8.  Managers  with  less  seniority  take  more  risks  than 
those  with  more  seniority, 

9.  Most  successful  executives  take  more  risks, 

10.  No  evidence  was  found  to  link  risk  propensity  and 
nationality  (15:247-267) . 

Research  indicates  that  risk  behavior  cannot  be 
predicted  solely  from  utility  theory  (7;  13;  15;  21) .  Other 
personal  and  situational  factors  also  influence  risk 
behavior.  Now  that  a  general  framework  of  risk  behavior  has 
been  established,  the  methods  for  studying  risk  propensity 
will  be  addressed. 
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studying  Risk 


The  classic  method  of  studying  risk  is  utility  theory. 

The  case  of  decision  making  under  risk  is  one  in 
which  considerable  consensus  exists  among  decision 
experts  as  to  how  one  ought  to  choose.  The  theory 
advises  which  alternative  to  select  in  con^jlex 
decision  situations  on  the  basis  of  one's  basic 
taste  and  preferences  about  risk  and  the  intrinsic 
value  of  the  attribute (s)  under  consideration. 

Both  of  these  elements  are  captured  through  a 
utility  function  so  that  maximizing  expected 
utility  becomes  the  guide  to  rational  behavior  in 
more  complex  situations  (21:11). 

Applying  utility  theory  should  make  most  decisions 

relatively  easy.  The  logical  choice  can  be  determined 

through  sound,  rational,  objective  reasoning.  To  illustrate 

this,  Schoemaker  presents  the  rational  man.  The  rational 

man  concept  views  an  ideal  person  (starting  with  Greek 

philosophy)  as  someone  leading  a  placid  life  amid  external 

turmoil  by  applying  reason  to  conduct  (21:2) .  The  idea  of 

the  rational  man  consists  of  three  fundamental  beliefs: 

1.  people  have  preference  structures  that  obey 
certain  axioms  of  well-behavedness  so  that 

2.  a  mathematical  representation  can  be  rendered 
of  these  preference  structures  and 

3 .  choice  can  be  modelled  as  maximizing  an 
imputed  objective  function  (e.g.,  expected 
utility)  subject  to  certain  economic 
constraints  (21:2) . 

Contrary  to  this  ideal  model,  it  has  been  shown  that 
decision  makers  do  not  make  choices  based  entirely  on 
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utility  (15;  17;  21) .  They  are  constantly  making  decisions 
that  violate  the  principles  of  the  expected  utility  theory 
(14:806) .  Edwards  and  Von  winterfeldt  cite  a  study  that 
demonstrates  this  behavior.  The  subjects  of  the  study  were 
given  the  task  of  choosing  between  two  lotteries.  The  first 
lottery  gave  them  the  opportunity  to  win  a  moderate  amount 
of  money  with  a  high  probability  of  success.  The  second 
lottery  gave  them  the  opportvinity  to  win  a  large  amount  of 
money,  but  with  a  low  probability  of  success.  The  subjects 
were  then  asked  to  bid  for  these  lotteries.  The  rational 
man,  when  applying  the  principles  of  expected  utility,  would 
pay  more  for  the  first  lottery.  Contrary  to  the  expected 
response,  most  subjects  were  willing  to  pay  more  for  the 
second  lottery,  violating  sotind,  rational,  objective 
reasoning  (6:664). 

Short  and  long  term  perspectives  are  other  factors  that 
influence  decisions  along  with  expected  utility.  "Risk 
affects  decision  making  such  that  choices  which  will  yield 
the  highest  expected  utility  in  the  long  term  are  often  not 
made"  (22:20).  Expected  utility  does  not  influence  long 
term  and  short  term  choices  in  the  same  fashion. 

Moore  concludes  that  when  a  particular  choice  will 
produce  a  success  70%  of  the  time,  the  30%  chance  of  failure 
may  be  insignificant  to  someone  who  will  make  the  choice 
once,  but  may  be  too  risky  for  a  decision  maker  who  will 
make  the  same  choice  several  times  (17:8).  The  result  is  a 
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focus  solely  on  the  probabilities  of  the  outcome  and  not  on 
sound,  rational,  objective  reasoning. 

Several  studies  have  been  conducted  on  risk  related 
behavior  amd  decision  making.  For  decisions  involving 
choice,  MacCrimmon  and  Wehrung  found  five  common 
generalities: 

1.  The  managers  showed  a  very  different 
willingness  to  take  risks  across  the 
standardized  situation. 

2 .  Managers  were  more  risk  averse  when  their  own 
money  was  at  stake  than  when  their  firm's 
resources  were  at  stake. 

3 .  Risk  taking  was  more  common  in  the  recognition 
and  adjustment  phases  of  risk  than  in  the 
choice  phase. 

4.  Managers  were  more  willing  to  take  risks  once 
in  a  risky  situation  than  in  entering  it. 

5.  A  manager's  willingness  to  take  risks  in  one 
standardized  situation  was  only  weakly  related 
to  his  willingness  to  take  risks  in  seemingly 
similar  situations. (15:269) 

Skibinski  conducted  a  study,  applying  these  ideas  to 
the  field  of  logistics.  He  surveyed  117  managers  and 
students  in  the  field  of  logistics.  His  study  concluded 
that  risk,  in  general,  significantly  affected  decisions.  He 
further  concluded  that  policy  addressing  risk  can  influence 
risk- inherent  decisions,  but  no  generalizations  can  be  made 
about  risk  policy.  Skibinski  finally  concluded  that: 
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The  implication  to  logistics  organizations  is  that 
a  risk  policy  which  is  stated  in  vague  or 
\incertain  terms  may  not  yield  the  desired  results. 
Management's  attempt  to  influence  risk- related 
decisions  may  only  be  successful  in  specific 
situations.  The  policy  will  likely  not  be  applied 
to  all  decision  varicd)les  uniformly.  Even  a  vague 
policy,  however,  will  reduce  the  level  of  risk 
taking  coir^ared  to  no  policy  at  all 
(22:104)  . 

Skibinski's  study  shows  that  policy  cannot  effectively 
control  how  the  individual  decision  makers  deal  with  risk. 
Policies  will  influence  decisions  to  certain  degrees,  but  no 
matter  how  much  direction  and  guidance  the  decision  maker  is 
given,  the  decision  is  unique  xinto  him.  This  is  even  more 
apparent  in  DoD  Systems  Acquisition. 


Risk  Management  in  the  DoD 

Harold  J.  Schutt,  Associate  Dean,  Department  of 
Research  and  Information,  Defense  Systems  Management  College 
(DSMC) ,  states  in  the  preface  of  Risk  Management  Concepts 
and  Guidance: 

There  are  no  "textbook"  answers  to  risk 
management.  Each  situation  is  different  and  each 
circumstance  requires  a  slightly  different 
approach"  (3: preface) . 

As  Schutt 's  statement  implies,  DoD  managers  are  given 
latitude  in  handling  risk.  Most  DoD  regulations  and 
policies  dealing  with  risk  provide  frameworks  from  which 
decision  makers  are  allowed  to  manage  risk.  These 
frameworks  mostly  consist  of  guidelines,  suggestions,  and 
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experiences  that  are  presented  to  help  managers  deal  with 
risk.  Specific  cuiswers  are  rarely  found  for  most 
situations. 

One  risk  management  approach  that  is  used  quite 
commonly  in  the  DoD  environment  is  to  classify  risk  into  its 
various  facets.  The  five  facets  are:  technical, 
programmatic,  supportability,  cost,  and  schedule  (3:3-3). 
Dividing  risk  into  these  five  facets  helps  in  "understanding 
the  source  of  the  risk  and  the  impact  area(s)  as  well  as 
providing  a  structure  to  examine  risk... if  the  risk  is  to  be 
managed  effectively"  (3:3-3) .  These  facets  surface  in  the 
guidelines  set  forth  in  Air  Force  Regulations  (APR)  70-15, 
Formal  Source  Selection  for  Major  Acquisitions,  and  APR  70- 
30,  Streamlined  Source  Selection  Procedures,  which  govern 
the  AF  source  selection  process. 

The  Air  Force  Source  Selection  Process  and  Risk  Assessment 

The  guidelines  for  managing  risk  in  the  source 
selection  process  address  the  functions  which  must  be 
assessed,  who  must  make  the  assessment,  and  when  the 
assessment  must  occur.  An  assessment  tool,  the  source 
selection  evaluation  matrix,  is  also  given. 

The  AF  requires  an  evaluation  of  risks  for  all 
proposals.  "The  risks  which  must  be  assessed  are  those 
associated  with  cost,  schedule,  and  performance  or  technical 
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aspects  of  the  program"  (4:17;  5:9) .  These  are  the  same 
facets  reccmsnended  in  the  DSMC  risk  handbook. 

AFR  70-15  and  AFR  70-30  State  that  the  risks  must  be 
formally  assessed  by  the  program  office,  the  offering 
contractor  and  the  Source  Selection  Evaluation  Board  (SSEB) 
or  Source  Selection  Evaluation  Team  (SSET) .  The  SSEB  and 
SSET  consist  of  personnel  who  advise  the  Source  Selection 
Authority  (SSA) .  The  primary  difference  between  these  two 
bodies  is  that  the  SSET  (a  streamlined  version  of  the  SSEB) 
is  designed  to  accelerate  the  source  selection  review 
process  (4;  5) .  For  the  purpose  of  this  research,  the  terms 
SSEB  and  SSET  can  be  used  interchangeably. 

The  program  office  will  perform  its  evaluation  of  risks 
based  upon  the  request  for  proposal  it  releases.  The 
program  office  is  required  to  submit  an  independent 
assessment  of  risks  to  the  SSEB  prior  to  the  receipt  of  any 
proposals.  The  offeror  is  required  to  assess  risks  with  the 
siobmission  of  the  proposal.  The  SSEB  will  make  its 
assessment  of  risk  based  upon  its  review  of  the  offerors' 
proposal  (4;  5) . 

The  guidelines  above  are  the  minimum  standard  for  risk 
assessment  during  the  source  selection  process.  More  risk 
assessment  may  occur  if  Air  Force  managers  deem  it 
appropriate . 

The  source  selection  evaluation  matrix  tool  is  a  rating 
technique  that  uses  color  codes  (Blue,  Green,  Yellow,  and 
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Red)  to  represent  the  technical  performance  cuid  the  terms 
High,  Moderate  and  Low  to  represent  the  assessed  risk  of  an 
evaluation  area.  The  definition  of  the  terms  are  given  in 
Table  2  and  Ted)le  3. 

The  use  of  these  categories  results  in  twelve  different 
possible  rcuikings  of  a  proposal.  Figure  l  shows  the 
possible  color -risk  combinations.  The  comparison  of  these 
ratings  in  an  area  will  help  the  SSA  make  decisions 
regarding  contract  award. 

erevious  Source  Selection  Risk  Related  Research 

To  date,  there  have  been  few  studies  of  risk  in  the  DoD 
source  selection  environment  (18;  20;  23) . 

Schenning  investigated  how  Air  Force  managers  evaluate 
risk  associated  with  various  program  decisions.  He 
concentrated  on  the  major  decisions  that  confront  most  mid¬ 
level  Air  Force  managers,  of  which  source  selection  was  one 
area  (20:8).  Schenning 's  research  found  that  personnel 
possessing  a  technical  degree  were  less  likely  than  those 
without  technical  degrees  to  rank  technical  risk  as  more 
important  them  cost  or  schedule  risk  in  a  source  selection 
environment  (20:78).  The  research  illustrates  how  other 
factors  (such  as  education)  can  influence  risk  behavior  in 
the  source  selection  environment. 


25 


Tcible  2 


Definition  of  Color  Rating  Codes  (4:16;  5:10) 


Color  Rating 

Definition 

Blue 

(Exceptional) 

Exceeds  specified  performance  or  capability 
in  a  beneficial  way  to  the  Air  Force;  and 
has  a  high  probability  of  satisfying  the 
requirement;  and  has  no  significant 
weakness. 

Green 

(Acceptable) 

Meets  evaluation  standards;  and  has  good 
probability  of  satisfying  the  requirement; 
and  any  weaknesses  can  be  readily  corrected. 

Yellow 

(Marginal) 

Fails  to  meet  evaluation  standards;  and  has 
low  probability  of  satisfying  the 
requirement;  euid  has  significant 
deficiencies  but  correctable. 

Red 

(Unaccept. ) 

Fails  to  meet  a  minimum  requirement;  and 
deficiency  requires  a  major  revision  to  the 
proposal  to  make  it  correct. 

Table  3 

Definition  of  Risk  Assessment  Codes  (4:16;  5:9) 


Risk 

Definition 

High  (H) 

Likely  to  cause  significant  serious  disruption 
of  schedule,  increase  in  cost,  or  degradation  of 
performance  even  with  special  contractor 
emphasis  and  close  government  monitoring. 

Moderate 

(M) 

Can  potentially  cause  some  disruption  of 
schedule,  increase  in  cost,  or  degradation  of 
performance.  However  special  contractor 
enphasls  or  close  government  monitoring  will 
probably  be  able  to  overcome  difficulties. 

Low  (L) 

Has  little  potential  to  cause  disruption  of 
schedule,  increase  in  cost,  or  degradation  of 
performance.  Normal  contractor  effort  and 
normal  government  monitoring  will  probably  be 
able  to  overcome  difficulties. 
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Yellow 

Red 

High 

Moderate 

Low 

Figure  i 

Technical  Performcuice/Rislc  Matrix 
(adapted  from  4;  5) 

Noffsirger's  research  looked  for  the  evaluation 

criteria,  used  in  past  AP  Logistics  Command  source 

selections  (base  level) ,  that  had  contributed  significantly 

to  the  award  decision  (I8:viii) .  He  performed  an  analysis 

on  historical  data  finding  relationships  among  award 

decisions,  risk  ratings,  color  ratings  emd  color- risk 

ratings  (18:54-69).  Noffsinger  found: 

Technical  risk  was  more  highly  correlated  with  the 
award  decision  than  the  actual  color  rating,  with 
lower  levels  of  risk  being  associated  with 
contract  award.  Combining  risk  and  color  ratings 
yielded  even  higher  correlation  (18:79) . 

Of  all  the  contracts  studied,  only  one  winning  proposal  was 

« 

not  rated  low  risk  (18:56).  Noffsinger  demonstrated  that 
the  offeror  evaluation  criteria  that  were  most  correlated 
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with  the  award  decision  were  those  that  dealt  with  risk, 
specifically  technical  risk.  The  research  does  not  address 
the  interrelationships  among  different  color-risk  ratings. 
For  example,  there  is  no  distinction  as  to  whether  a  Green - 
High  rating  is  prefercible  to  a  Yellow-Low  rating. 

Thomas  examined  risk  assessment  and  scoring  in  the  DoO 
environment  in  order  to  develop  a  scoring  model  (23) .  The 
scoring  model  was  intended  to  provide  a  reliable  and 
standardized  methodology  of  deriving  area  level  scores  from 
item  and  factor  level  assessments  (23:235).  The  model  was 
constructed  using  policy  guidance  and  the  results  of  a  fact 
finding  questionnaire.  The  questionnaire  asked  69  AF 
members  at  Aeronautical  Systems  Division,  who  were 
experienced  in  source  selections,  if  the  present  scoring 
system  was  adequate.  Inputs  were  solicited  from  members  who 
stated  the  present  system  needed  improvements  (23:77).  The 
results  of  Thomas'  work  resulted  in  a  model  that  conformed 
to  all  the  regulations  (at  that  time)  and  was  an  improvement 
over  existing  scoring  models  (23:234-238). 

The  cdjove  research  examined  how  risk  is  evaluated  in 
program  decisions,  how  risk  has  contributed  to  the  source 
selection  decision  and  how  risk  can  be  modeled  to  provide 
consistent  proposal  scorings  in  the  source  selection 
environment.  None  of  the  studies  examined  how  technical 
color- risk  ratings  are  traded-off  by  source  selection  team 
members . 
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S\jinmarY 

This  literature  review  has  estcUalished  a  working 
definition  of  risk  by  presenting  the  common  features  from 
the  differing  vie%»points  examined.  The  results  of  research 
studying  behavior  of  managers  when  confronting  risks  were 
identified.  Utility  theory  was  used  to  demonstrate  that  in 
many  circumstances  optimal  choices  are  obtained  through 
sound,  rational,  objective  reasoning.  Studies  have  shown 
that  in  spite  of  utility  theory  "truths,"  people  do  not 
select  the  expected  choice  predicted  by  utility  theory.  The 
affect  of  risk  on  these  choices  in  the  source  selection 
environment  needs  to  be  studied. 

DoD  and  the  AF  have  established  policies  concerning 
risk  management  in  the  source  selection  environment.  The 
guidance  is  intended  to  assist  decision  makers  in  managing 
the  risks  involved  in  the  acquisition  process.  Studies  have 
shown  that  policies  have  some  influence  on  the  decisions 
made  by  managers,  but  they  do  not  determine  the  final 
decision. 

None  of  the  previous  source  selection  research  studies 
addresses  the  risk  propensity  of  source  selection  decision 
makers  and  how  their  propensity  affects  their  decisions. 
Previous  research  efforts  focus  on  historical  analysis  of 
source  selection  decisions.  This  research  provides  a  better 
understanding  of  risk  management  in  the  source  selection 
process  by  examining  the  risk  propensity  of  the  source 
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selection  participants  using  the  color-risk  assessment  tool 
recoimnended  by  AFR  70-15  and  AFR  70-30. 
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III.  Methodology 


Introduction 

This  chapter  explains  the  methods  used  to  answer  the 
investigative  questions  posed  in  Chapter  I  in  support  of  the 
research  objective.  Determination  of  the  population  and 
sample  size,  development  of  the  survey  instrioment, 
reliability  and  validity  issues  of  the  research,  and  the 
methods  of  analyzing  the  data  are  discussed.  The  chapter 
concludes  by  addressing  how  the  investigative  questions  have 
been  addressed. 

Population 

To  accomplish  the  objective  of  this  research,  the  study 
focused  on  individuals  who  had  participated  in  a  source 
selection.  Specifically,  the  target  population  was  Air 
Force  (AF)  personnel  (military  and  civilian)  who  have 
participated  in  a  source  selection.  The  population 
consisted  of  acquisition  personnel  from  all  Air  Force 
Product  and  Materiel  Centers.  The  target  population  is 
constantly  changing  as  personnel  are  entering  and  leaving 
the  acquisition  work  force  through  assignment  changes,  as 
well  as  AF  accessions  and  attrition.  The  population 
included  all  officers  and  civilians  who  have  participated  in 
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at  least  one  source  selection  ^nd  are  currently  working  in 
the  acquisition  career  field. 

Sampling 

The  sample  representing  the  population  of  interest  was 
AF  acquisition  personnel  who  have  participated  in  a  source 
selection  at  Wright -Patterson  Air  Force  Base  OH  (WPAFB) , 
Aeronautical  Systems  Center  (ASC) .  The  sample  was  further 
constrained  to  personnel  that  were  still  assigned  to  WPAFB 
at  the  time  of  survey  mailing. 

The  sample  of  personnel  at  ASC  is  representative  of  the 
other  product  centers .  Acquisition  personnel  generally  have 
experience  at  more  than  one  product  center  due  to  the  nature 
of  Department  of  Defense  positions.  This  is  especially  true 
for  military  personnel.  Also,  personnel  at  the  product 
centers  have  similar  educational  backgrounds.  The  personnel 
also  use  the  same  regulations  and  military  standards  in 
conducting  business.  Therefore,  as  was  shown  in  the 
literature  review,  the  aggregate  attitudes  and  perceptions 
of  acquisition  personnel  at  the  various  product  centers 
should  be  similar  when  dealing  with  risk. 

These  personnel  were  identified  by  the  Request  For 
Proposal  (RFP)  and  Source  Selection  Support  Program  office 
located  at  WPAFB.  This  office  maintains  records  on 
participants  in  ASC  source  selections,  but  information 
regarding  personnel  reassignments  is  not  maintained  with 
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this  data.  To  increase  the  response  rate,  only  participants 
in  recent  source  selections  were  surveyed.  They  identified 
202  personnel  who  met  the  survey  requirements.  Due  to  the 
dynamic  nature  of  the  acquisition  work  force,  the  sanple 
represents  a  moment  in  time. 

Survey  Development 

A  mail  survey  was  selected  as  the  best  method  to  meet 
the  research  objective.  The  survey  provided  many  advantages 
over  observational  and  experimental  methods. 

Surveys  tend  to  be  more  economical  than  observation  and 
experimentation  (8:318).  In  measuring  attitudes  and 
decisj.on  processes,  observation  and  experimentation  would 
have  required  more  time  than  was  available  to  yield  an 
adequate  sample  size.  Observational  data  is  also  limited  in 
the  fact  that  attitudes  and  decision  processes  are  not 
always  visually  detectcUale. 

The  survey  consisted  of  four  sections:  I)  individual 
risk  propensity,  II)  source  selection  evaluation  matrix 
color-risk  rankings.  III)  source  selection  evaluation  color- 
risk  pairwise  comparisons,  and  IV)  background  information. 

A  copy  of  the  survey  is  contained  in  Appendix  A. 

Section  I  measured  individual  risk  propensity  in  terms 
of  choices  related  to  monetary  gains  and  losses.  These 
questions  were  used  to  categorize  individuals  as  risk 
averter,  risk  neutral  and  risk  taker.  The  questions  were 
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based  on  survey  questions  used  by  Bmmelhainz,  Kahnenicui  & 
Tversky,  and  Skibinski  (7;  13;  22) . 

The  original  survey  questions  presented  two  choices 
with  different  risks  and  different  expected  values.  For 
exanple,  individuals  were  asked  if  they  preferred: 

A.  An  80%  chance  of  gaining  $4,000  euid  a  20% 
chance  of  gaining  $0,  or 

B.  A  certain  gain  of  $3,000  (22:147) . 

A  risk  taker  would  prefer  the  first  choice  while  a  risk 
averter  would  prefer  the  second  choice.  There  was  no  risk 
neutral  choice. 

In  the  survey  for  this  research,  three  different 
choices  were  provided.  The  first  two  choices  involved 
different  probabilities  of  outcome  and  monetary  values,  but 
had  the  same  expected  value.  The  third  choice  was  for  no 
preference  between  the  first  two  choices.  For  example, 
individuals  were  asked  if  they  preferred: 

A.  An  80%  chance  of  gaining  $4,000  cuid  a  20% 
chance  of  gaining  $0. 

B.  A  certain  gain  of  $3,200. 

C.  No  preference. 

A  risk  taker  would  prefer  the  first  choice  because  there  is 
a  chance  of  gaining  more  money  with  the  potential  loss  of 
gaining  nothing.  A  risk  averter  would  choose  the  second 
choice  because  there  is  no  risk.  A  risk  neutral  would  see 
that  the  expected  values  are  the  same  and  have  no 
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preference.  TcUole  4  shows  the  six  questions  used  in  the 
survey  and  how  the  responses  are  associated  with  risk 
propensity. 


Table  4 

Survey  Section  I: 

Risk  Propensity  Response  Categorization 


Question  Risk  Averter  Risk  Neutral  Risk  Taker 


1 

Certain  gain 
of  $3200 

No  preference 

80%  chance  of 
$4000  gain, 

20%  chance  of 
$0  gain 

2 

25%  chance  of 
$3200  gain, 

75%  chance  of 
$0  gain 

No  preference 

20%  chance  of 
$4000  gain, 

80%  chance  of 
$0  gain 

3 

Certain  loss 
of  $3200 

No  preference 

80%  chance  of 
$4000  loss, 

20%  chance  of 
$0  loss 

4 

25%  chance  of 
$3200  loss, 

75%  chance  of 
$0  loss 

No  preference 

20%  chance  of 
$4000  loss, 

80%  chance  of 
$0  loss 

5 

90%  chance  of 
$3000  gain, 

10%  chance  of 
$0  gain 

No  preference 

45%  chance  of 
$6000  gain, 

55%  chance  of 
$0  gain 

6 

90%  chance  of 
$3000  loss, 

10%  chance  of 
$0  loss 

No  preference 

45%  chance  of 
$6000  loss, 

55%  chance  of 
$0  loss 

Section  ll  presented  definitions  of  color  and  risk 
ratings  used  in  source  selections.  The  section  established 
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how  individuals  trade-off  performance  and  risk  in  their 
decision  processes.  Respondents  were  asked  to  rank  the 
nine  combinations  of  risk  ratings  (High,  Moderate,  Low)  and 
color  ratings  (Blue,  Green,  Yellow)  from  l  (most  preferred) 
to  9  (least  preferred) .  Figure  2  illustrates  a  possible 
ranking  from  a  respondent.  The  color  rating  of  Red  was 
omitted  because  any  proposal  with  this  rating  is  not 
eligible  for  contract  award  per  APR  70-15  and  APR  70-30. 

Section  II  also  compared  rankings  between  item  summary 
levels.  Respondents  were  asked  to  ccmplete  a  matrix  for  two 
different  item  summary  levels;  technical  capability  and 
management . 


Figure  2 

Survey  Section  ll:  Exart^le  of 
Completed  Performance/Risk  Matrix 
(adapted  from  4;  5) 
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Section  III  provided  pairwise  comparisons  of 
preferences  between  selected  color -risk  combinations  from 
the  source  selection  evaluation  matrix.  Respondents  were 
asked  to  indicate  their  preferences  for  color -risk  pairs 
using  eui  ordinal  scale.  Figure  3  illustrates  the  usage  of 
the  ordinal  scale  in  Section  in.  Selecting  a  i  or  a  7 
indicates  the  strongest  preference  for  Green-Low  or  Blue- 
MOderate  respectively.  Selecting  4  indicates  that  there  is 
no  preference. 

Pairwise  comparisons  provided  cuiother  method  to 
estcdslish  the  rank  order  structure.  It  also  established  the 
degree  of  preference  between  color-risk  combinations. 

Color:  Green  _  Color:  Blue 

Risk:  Low  1  2  3  4  5  6  7  Risk:  Moderate 

Figure  3 

Ordinal  Scale  Pairwise  Comparison  of  Color -Risk  Ratings 

Section  IV  provided  background  information  for  the 
demographics  of  the  san^le.  The  respondents  were  asked 
their  military  rank/civilian  grade,  years  of  acquisition 
experience,  acquisition  specialty  (program  management, 
logistics,  program  control,  etc.)  and  the  highest  position 
held  in  a  source  selection.  They  were  also  asked  to  assess 
their  risk  propensity  in  their  personal  cuid  professional 


lives.  The  demographic  questions  also  provided  a  means  to 
confirm  that  survey  respondents  met  the  san^le's 
requirements . 

Mfethod  of  Analysis 

Risk  Propensity  Categorization.  Individuals  were 
categorized  into  risk  propensity  groups  based  on  the  results 
of  Section  I  of  the  survey.  The  respondents  were  given  a 
score  of  +l  if  they  chose  the  riskier  option.  A  score  of  -l 
was  given  for  selecting  the  less  risky  option.  When  no 
preference  was  selected,  a  score  of  0  was  given.  All  the 
scores  for  the  questions  were  then  summed.  A  negative  total 
indicated  that  the  respondent  was  a  risk  averter.  A 
positive  total  indicated  that  the  respondent  was  a  risk 
taker.  A  total  of  zero  indicated  that  the  respondent  was 
risk  neutral. 

Performance/Risk  Trade-Off.  Sections  II  and  III  were 
used  to  answer  how  source  selection  participeuits  trade-off 
performance  versus  risk.  Section  ll  was  used  to  establish - 
rank  orders  among  the  various  color- risk  ratings  of  the 
source  selection  evaluation  matrix.  Section  III  was  used  to 
estcd^lish  rank  orders  by  indicating  preferences  between 
color-risk  rating  pairs.  The  rank  orders  were  estcdilished 
for  the  entire  sanple. 
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Rank  orders  were  established  by  using  the  Friedman  test 
(1:  299) .  The  two  assunptions  of  this  test  are: 

1.  the  b  k-variate  random  variables  are  mutually 
independent 

2.  within  each  block  the  observations  may  be 
rcuiked  according  to  some  criterion  of  interest 
(1:  296  and  299) 

By  the  definition  of  the  source  selection  evaluation  matrix, 
each  of  the  possible  rankings  is  mutually  independent,  thus 
satisfying  assumption  one.  In  the  test,  b  is  the  number  of 
respondents  and  k  is  the  nxmnber  of  possible  ranks  of  the 
color- risk  ratings  (nine) .  Assunption  two  is  the  purpose  of 
Section  II,  ranking  the  source  selection  evaluation  matrix 
combinations. 

The  null  hypothesis  for  the  Friedman  test  is  that  "each 
ranking  of  the  random  variables  within  a  block  is  equally 
likely"  (1:296) .  The  alternate  hypothesis  is  that  at  least 
one  of  the  random  varicUsles  yields  larger  observed  ranks 
(1:297) .  Acceptance  of  the  null  hypothesis  was  based  on  a 
level  of  significance  (a)  equal  to  0.05. 

In  addition  to  establishing  rank  order,  the  Friedman 
test-  also  ested>lishes  the  significance  between  individual 
rankings  so  conclusions  may  be  drawn.  These  decisions  could 
include  that  rank  l  and  reuik  2  are  statistically  different 
and  must  fall  in  that  rank  order.  Another  decision  could  be 
rank  l,  rank  2,  and  rauik  3  are  not  statistically  different 
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from  one  another,  but  all  are  statistically  different  from 
rank  4. 

Section  III  required  respondents  to  indicate 
preferences  according  to  «m  ordinal  scale.  Preference 
between  color- risk  rating  pairs  was  identified  by 
examination  of  the  median  value.  Referring  to  Figure  3,  a 
median  value  of  2  indicates  that  a  Green- Low  color  rating  is 
ranked  higher  in  the  overall  rank  order  structure  than  the 
Blue -Moderate  color  rating.  The  preferences  of  the  color - 
risk  ratings  were  used  to  build  a  rank  order.  The  reuik 
order  established  by  the  ordinal  scale  was  compared  to  the 
Friedman  test  rank  order  results  to  establish  reliability. 

Risk  Propensity  and  Trade-Off  Decisions.  Using  the 
risk  propensity  categories  previously  established  in  the 
analysis  of  Section  I  of  the  survey,  rank  order  structures 
were  developed  for  the  different  risk  propensity  categories. 
The  rank  order  structures  were  developed  according  to  the 
Friedman  test  described  above.  The  rank  order  structures 
were  then  compared  among  risk  propensity  categories  using 
Spearman  rank  correlations.  Conparisons  were  also  made  with 
the  rank  order  structure  of  the  entire  sanqple. 

The  Spearman  rank  correlation  coefficient  is  a 
statistic  that  tests  for  independence  between  two  random 
variables  (1:254) .  The  null  hypothesis  for  the  test  is  that 
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the  random  variables  are  Independent.  The  alternate 
hypothesis  is  that  the  variables  are  correlated  (1:254). 

The  results  will  indicate  differences  among  rank  order 
structures  defined  by  the  risk  propensity  groups  (ie.  does 
risk  propensity  affect  rank  order) .  The  Friedman  rank  sums 
were  used  for  the  Spearman  comparisons. 

Item  Sximroary  and  Trade-Off  Decisions.  Section  II 
required  respondents  to  estcUolish  rank  orders  for  two 
different  item  sximmary  levels:  technical  capability  and 
management.  Friedman  tests  were  used  to  estad^lish  rank 
orders  by  item  summary  level  for  the  entire  sample.  The 
Spearman  rank  correlation  was  conducted  to  compare  the  rank 
orders . 

Degree  of  Preference.  Section  III  required  respondents 
to  indicate  degree  of.  preferences  for  color- risk  rating 
pairs  according  to  a  seven  point  ordinal  scale.  Degree  of 
preference  between  color-risk  rating  pairs  was  identified  by 
examination  of  the  median  value.  A  median  value  of  4 
indicated  no  preference  for  either  color -risk  rating.  A 
median  value  of  1  or  7  indicated  a  strong  preference  for  one 
of  the  color- risk  ratings.  Median  values  of  2  and  6 
indicated  a  moderate  preference  for  one  of  the  color- risk 
ratings,  while  a  3  or  5  indicated  a  slight  preference  for 
one  of  the  color -risk  ratings. 
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Strategies  for  Reliability 


Reliability  focuses  on  determining  the  randomness  or 
unstable  error  of  the  measurement  device  (8) .  To  ensure  the 
survey  was  relicdjle,  the  following  strategies  were  used: 

1.  standardized  definitions  of  color  and  risk 
ratings 

2 .  two  methods  to  determine  rank  order 

3.  adoption  of  risk  propensity  questionnaire  used 
in  three  previous  studies  (7;  13;  22) 

4.  comparison  of  risk  propensity  category 
assignment  and  respondent  self  evaluation  of 
risk  propensity 

5.  pretest  of  survey  instrument 

The  definitions  of  color  and  risk  ratings  used  in 
Sections  II  and  III  of  the  survey  were  taken  directly  from 
the  source  selection  governing  regulations,  APR  70-15  and 
APR  70-30.  Since  the  target  sample  had  participated  in  a 
source  selection,  these  terms  (as  well  as  the  use  of  an 
evaluation  matrix)  were  familiar  to  the  respondents.  Random 
errors  due  to  misrepresentation  of  actual  terminology  used 
in  the  source  selection  environment  have  been  eliminated  by 
providing  current  definitions.  The  definitions  used  are 
valid  until  the  regulations  governing  the  source  selection 
ranking  process  change. 

Reliability  was  also  addressed  through  the  design  of 
the  questionnaire.  Sections  II  and  III  require  the 
respondent  to  discriminate  by  preferences  among  color- risk 
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ratings.  Con^aring  the  results  of  these  sections  should 
reveal  similar  rank  orders.  For  exait^le,  if  a  Blue -Low  risk 
was  ranked  second  and  a  Green -High  risk  was  ranked  fifth, 
Section  III  should  also  show  that  in  a  pairwise  comparison 
the  Blue-Low  risk  was  the  preferred  color-risk  rating. 
Results  of  constructing  the  rank  orderisi:  in  these  two 
different  manners  demonstrated  internal  consistency  of  the 
suirvey.  A  Spearman  rank  correlation  of  0.9958  was  found 
when  comparing  the  two  rank  orders. 

As  previously  stated.  Section  I,  risk  propensity 
categorization,  was  derived  from  surveys  of  other  risk 
propensity  studies  (7;  13;  22) .  Meta-analysis  of  these 
studies  found  that  the  results  obtained  are  replicable 
(22:94-95) . 

The  survey  was  pretested  during  the  development  stage. 
Pretesting  occurred  in  two  steps.  The  first  step  was  to 
administer  the  draft  survey  to  five  people  who  were  familiar 
with  survey  development.  The  participants  completed  the 
draft  survey  in  the  presence  of  the  research  team. 
Participants  were  encouraged  to  ask  questions  and  provide 
criticism  of  the  draft  survey.  This  step  was  intended  to 
eliminate  weaknesses  in  the  measurement  technique  and 
ambiguity  in  the  instructions  and  terminology.  The  feedback 
from  this  step  was  incorporated  into  another  draft  survey 
for  the  second  step  of  pretesting. 
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The  second  step  of  pretesting  consisted  of 
atoinistering  the  survey  to  seven  Air  Force  Institute  of 
Technology  students  and  three  personnel  in  the  RFP  and 
Source  Selection  Support  Program  Office  who  met  the  survey 
population's  requirements  and  were  familiar  with  survey 
development.  These  people  were  given  the  survey  without  any 
instructions  other  than  to  follow  the  directions  of  the 
survey  and  provide  feedback  to  the  research  team.  This  step 
was  intended  to  test  the  survey  in  as  realistic  environment 
as  possible  and  to  determine  the  time  required  to  complete 
the  survey.  The  second  pretest  step  also  assured 
elimination  of  measurement  weaknesses  and  ambiguities  in  the 
instructions  and  terminology  of  the  survey. 

Validity 

"Validity  is  the  extent  to  which  differences  found  with 
a  measuring  tool  reflect  true  differences  among  those  being 
tested"  (8:180).  The  validity  of  Section  I  was  established 
by  using  a  previously  accepted  survey  instniment.  Validity 
of  Sections  ll  and  III  was  established  through  the  use  of 
the  actual  terminology  and  evaluation  methods  used  in  the 
source  selection  process.  Respondents  are  faced  with  the 
same  trade-off  decisions  when  actually  participating  in  a 
source  selection. 
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The  purpose  of  this  chapter  was  to  provide  the 
methodology  used  to  answer  the  investigative  questions 


supporting  the  research  objective.  Table  5  describes  the 
methods  used  to  answer  each  investigative  question.  This 
chapter  has  discussed  the  methodology  of  collecting, 
organizing  and  analyzing  the  data  used  to  accomplish  the 
research  objective  addressed  by  this  thesis.  The  next 
chapter  reports  the  results  of  the  study. 


Table  5 

Method  of  Answering  Investigative  Questions 


Investigative  Questions  (IQ) 

Methods  Used  to  Answer  IQ 

1)  What  is  the  risk 
propensity  of  the  source 
selection  participants? 

1)  Section  I  of  the  survey 
was  used. 

2)  How  do  source  selection 
participants  trade-off 
performance  versus  risk? 

2)  Section  I I  and  III  of 
the  survey  were  used. 

(Friedman  test,  median 
values) 

3)  Does  risk  propensity 
influence  the  trade-off 
decision? 

3)  Sections  II  &  III  of 
survey  were  used.  (Results 
of  IQ2,  Spearman  rank  corr.) 

4)  Does  the  performance 
versus  risk  trade-off 
decision  vary  based  on  the 
item  being  evaluated? 

4)  Section  II  of  the  survey 
was  used.  (Results  of  IQ2  & 
IQ3 ,  Spearman  rank  corr . ) 

5)  To  what  degree  are 
color-risk  ratings  preferred 
over  one  anoth> 

5)  Section  III  of  the 
survey  was  used.  (Median 
values) 
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This  chapter  presents  the  results  of  the  analysis  of 
the  survey  data.  The  chapter  describes  the  sairple  surveyed, 
explains  why  some  responses  were  not  used  in  the  analysis, 
and  the  demographics  of  the  respondents.  The  results  are 
then  presented  for  each  applicable  investigative  question. 
The  statistical  results  associated  with  each  question  are 
presented. 

Survey  Target  Sample  and  Responses 

The  target  sample  was  personnel  currently  stationed  at 
Wright -Patterson  Air  Force  Base  OH  (WPAFB)  Aeronautical 
Systems  Center  (ASC)  who  had  participated  in  a  source 
selection.  The  sample  of  202  people  was  identified  by  the 
Request  For  Proposal  (RFP)  and  Source  Selection  Support 
Program  Office  at  wpafb. 

Of  the  202  surveys  mailed,  80  surveys  were  returned  by 
the  deadline.  Seven  of  the  202  surveys  mailed  were  returned 
unopened  due  to  inaccurate  mailing  addresses  or  due  to 
transfer  of  personnel.  Of  the  80  surveys  returned,  four 
were  unusable  because  the  survey  had  not  been  filled  out 
completely.  The  resulting  data  set  consisted  of  76 
returned,  correctly  filled-out  surveys.  This  is  a  response 
rate  of  40%. 


Figures  4,  5  and  6  show  the  demographic  information 
collected  on  the  responding  individuals.  The  data  indicates 


that  the  majority  of  respondents  had  participated  in  the 
source  selection  as  an  item  captain,  area  chief,  or 
evaluator  (8i.€%) .  The  largest  responding  group  were  those 
presently  working  as  progreun  meuiagers  (32.9%)  .  Over  half  of 
the  respondents  were  lower  level  managers  or  workers  (O-i  to 
0-3,  6S-12  euid  below) .  The  remaining  group  was  primarily 
middle  level  managers  (0-4  and  0-5,  GS-13  to  GM-14) .  The 
average  acquisition  experience  of  the  respondents  was  ii.O 
years.  The  range  of  experience  was  from  a  minimum  of  l  year 
to  a  maximum  of  34  years.  Based  on  the  demographic 
characteristics  of  the  san^le,  the  respondents  are 
reflective  of  the  population. 


Other  SSA 

(7.9%)  (2.6%) 


Figure  4 

Highest  Position  Held  in  a  Source  Selection 


inyestiaative  Questions 


1.  What  is  the  risk  propensity  of  the  source 
selection  participants? 

2.  How  do  source  selection  participants  trade-off 
performance  versus  risk? 

3.  Does  risk  propensity  influence  the  trade-off 
decision? 


4.  Does  the  performance  versus  risk  trade-off 
decision  vary  based  on  the  item  being 
evaluated? 

5.  To  what  degree  are  color- risk  ratings 
preferred  over  one  another? 


Other 

(13.2%) 


Logistics 

(11.8%) 


Contracting 

(9.2%) 


Engineering 
(18.4%) 


Program 

Control 

(14.5%) 

Figure  5 

Acquisition  Specialty 


Program 

Management 

(32.9%) 
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0-6  and  above 
and 


GM-15and  above 


(3.9%) 


GS-13toGM-14 

(40.8%) 


GS-12  and  under 
(55.3%) 


Figure  6 

Military  Reuik  euid  Civilian  Grade 


Risk  Propensity  Categorization.  What  Is  the  risk 
propensity  of  the  source  selection  participants  s«q>led7 
The  assessment  in  survey  Section  I  resulted  in  the 
identification  of  23  risk  averters,  29  risk  neutrals,  and  24 
risk  takers.  The  assessment  scale  ranged  in  integers  from 
-6  to  6.  A  score  of  zero  resulted  in  a  risk  propensity 
assignment  of  risk  neutral.  A  negative  score  resulted  in  a 
risk  propensity  assignment  of  risk  averter,  while  a  positive 
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Frequency 


score  resulted  in  a  risk  propensity  assignment  of  risk 
taker.  The  histogram  in  Figure  7  shows  the  distribution  of 
raw  scores  for  the  entire  samiple. 


Raw  Score  Totals 


y//\  Risk  Averters  |  |  Risk  Neutrals  jj^j^  Risk  Takers 


Figure  7 

Risk  Propensity  Distribution 


50 


Performance/Risk  Trade-Off.  How  do  sourca  aelactlon 
participants  trada-off  parfoxmanca  ▼arsus  risk?  Figure  8 
shows  the  results  of  color-risk  rating  ranks. 


Blue 

Green 

Yellow 

High 

5 

7 

9 

Moderate 

2 (tie) 

4 

8 

Low 

1 

2 (tie) 

6 

Figure  8 

Color-Risk  Rating  Ranks 

Analysis  shows  that  a  definite  rank  order  exists  among 
the  color-risk  ratings  (a =  0.05).  The  relationships  shown 
above  affirm  that  lower  risks  are  preferred  to  higher  risks 
and  blue  ratings  are  preferred  to  green  ratings,  which  are 
preferred  to  yellow  ratings.  The  trend  is  to  prefer  color- 
risk  ratings  in  the  lower  left  hand  corner  of  the  source 
selection  evaluation  matrix.  The  Blue -Moderate  and  Green - 

Low  color- risk  ratings  were  not  statistically  different  from 

% 

one  another  (a  =  0.05).  Tedale  6  shows  the  results  of  the 
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Table  6 


Friedman  Test  Color-Risk  Rankings 
for  The  Entire  Sample,  Technical  Capability  Item  Summary 
(Note:  rankings  that  are  not  significcuitly 
different  are  identified  by  ***) 


1) 

Blue  Low 

76 

2t) 

Blue  Moderate*** 

199 

2t) 

Green  Low*** 

204 

4) 

Green  Moderate 

323 

5) 

Blue  High 

395 

6) 

Yellow  Low 

460 

7) 

Green  High 

512 

8) 

Yellow  tfoderate 

567 

9) 

Yellow  High 

682 

Level  Discriminator 

(a  =  0.05) :  22.08 

Friedman  test.  Appendix  B  contains  the  formulas  and 
calculations  used  in  the  Friedman's  test. 

A  rank  order  of  color- risk  ratings  for  the  entire 
sample  for  the  technical  capability  item  summary  was 
constructed  using  an  ordinal  scale  of  pairwise  conparisons. 
The  results  are  shown  in  Table  7.  Table  8  shows  the 
preferences  between  color -risk  rating  pairs  used  to 
construct  the  rank  order.  Comparison  of  the  ordinal  scale 
rank  order  to  the  Friedman  test  rank  order  show  that  the 
relative  order  among  color-risk  ratings  is  the  same.  The 
only  exception  is  that  the  Yellow- Low  and  Green-High  ratings 
are  tied.  The  Spearman's  rank  correlation  indicated  that 
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Tcible  7 


Ordinal  Scale  Color- Risk  Reuikings 
for  The  Entire  San?)le,  Technical  Capability  Item  Stunmary 


1)  Blue  Low 
2t)  Blue  Moderate 
2t)  Green  Low 

4)  Green  Moderate 

5)  Blue  High 
6t)  Yellow  Low 
6t)  Green  High 

8)  Yellow  Moderate 

9)  Yellow  High 


Table  8 

Color -Risk  Pairing  Preferences 
for  the  Entire  Sanple 


Relationship 

Preference 

tfedian 

Blue  High  -  Green  Moderate 

Green  Moderate 

5 

Green  Low  -  Blue  High 

Green  Low 

2 

Blue  High  -  Yellow  Low 

Blue  High 

3 

Yellow  Moderate  -  Green  High 

Green  High 

5 

Yellow  Low  -  Green  High 

NONE 

4 

Blue  Moderate  -  Yellow  low 

Blue  Moderate 

2 

Yellow  Low  -  Green  Moderate 

Green  Moderate 

5 

Green  Low  -  Blue  Moderate 

NONE 

4 

Blue  High  -  Yellow  Moderate 

Blue  High 

3 

the  rcuikings  were  statistically  the  same  (r  =  0.9958)  . 
Arriving  at  the  same  rank  order  using  two  different  methods 
adds  to  the  relieibility  of  establishing  the  true  rank  order 
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of  color- risk  ratings  used  in  the  source  selection 
evaluation  matrix. 

Risk  Propensity  and  Trade-Off  Decisions.  Does  risk 
propensity  Influence  the  trade-off  decision?  Risk 
propensity  does  not  influence  the  trade-off  decision.  Table 
9  shows  the  reink  orders  established  by  risk  propensity  for 
the  technical  capability  item  svimmary  using  the  Friedman 
test.  The  color-risk  rank  order  is  identical  for  risk 
averters  and  risk  neutrals.  Risk  teUcers'  rank  order  differs 
from  the  other  two  categories  only  in  that  there  is 
difference  between  the  Yellow- Low  and  Green-High  ratings. 
Appendix  B  contains  the  formulas,  calculations,  and  the 
level  discriminator  used  in  the  Priednan’s  test  to  establish 
rank  orders  for  all  risk  propensity  categories. 

Spearman  rank  correlations  were  calculated  for  all  the 
combinations  of  risk  propensity.  The  result  of  the  Spearman 
rank  correlations  is  that  there  is  a  high  degree  of 
correlation  among  risk  propensity  categories.  Table  10 
shows  the  Spearman  rank  correlation  values. 

Item  S^  ary  and  Trade-Off  Decisions.  Does  the 

perfoxaance  versus  risk  trade~off  decision  vary  based  on  the 
item  being  evaluated?  The  trade-off  decision  based  on 
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TcJale  9 


Color- Risk  Ranking  by  Risk  Propensity, 
Technical  Capability  Item  Sxjiranary 


Risk  Averter 

Risk  Neutral 

Risk  Taker 

1)  Blue  Low 

2t)  Blue  Ifoderate 
2t)  Green  Low 

4 )  Green  Ifoderate 

5 )  Blue  High 

6)  Yellow  Low 

7 )  Green  High 

8)  Yellow 

Moderate 

9)  Yellow  High 

1)  Blue  Low 

2t)  Blue  Moderate 
2t)  Green  Low 

4)  Green  Moderate 

5 )  Blue  High 

6)  Yellow  Low 

7 )  Green  High 

8)  Yellow 

Moderate 

9)  Yellow  High 

1)  Blue  Low 

2t)  Blue  Moderate 
2t)  Green  Low 

4)  Green  Moderate 

5)  Blue  High 

6 1).  Yellow  Low 

6t)  Green  High 

8)  Yellow 

Moderate 

9)  Yellow  High 

Table  10 

Spearman  Rank  Correlations  for  Risk  Propensity  Categories, 
Technical  Capability  Item  Summary 


Risk  Averter 

Risk  Neutral 

Risk  Neutral 

0.9833 

N/A 

Risk  Taker 

1.0000 

0.9833 

performance  versus  risk  does  not  vary  according  to  item 
summary  level . 

Table  11  shows  the  results  of  the  Friedman  test  color - 
risk  rankings  by  summary  areas.  The  Spearman  rank 
correlation  for  item  suiranazy  levels  is  l.OOOO.  The  rank 
order  is  exactly  the  same  for  each  item  siunmary  level . 
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Table  11 


Comparison  of  Item  Summary  Levels 
on  Color -Risk  Rankings  Using  the  Friedman  Test 
(Note:  rankings  that  are  not  significantly 
different  are  identified  by  ***) 


Technical  Capability: 

Management : 

Blue  Low 

76 

Blue  Low 

76 

Blue  ttoderate*** 

199 

Green  Low*** 

201 

Green  Low*** 

204 

Blue  Moderate*** 

207 

Green  tfoderate 

323 

Green  Moderate 

334 

Blue  High 

395 

Blue  High 

394 

Yellow  Low 

460 

Yellow  Low 

447 

Green  High 

512 

Green  High 

516 

Yellow  Moderate 

567 

yellow  Moderate 

561 

Yellow  High 

682 

Yellow  High 

684 

Level  Discriminator 

Level  Discriminator 

(g  =  0.05)  : 

22.08 

(a  =  0.05) : 

22.99 

Degree  of  Preference.  To  wbat  degree  are  color* risk 
ratings  preferred  over  one  another?  Table  12  shows  the 
preference  among  various  color-risk  rating  pairs.  No 
preference  exists  between  the  comparison  of  Green- Low  to 
Blue-Moderate  ratings  and  Yellow- Low  to  Green-High  ratings. 
The  greatest  difference  in  preferences  occur  when  comparing 
the  Green-Low  to  Blue-High  ratings  and  the  Blue-Moderate  to 
Yellow- Low  ratings.  These  comparisons  showed  moderate 
preferences.  The  remaining  five  con^jarisons  showed  only 
slight  preferences  among  the  color-risk  pairs.  Histograms 
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of  the  responses  for  color -risk  preferences  are  provided  in 
Appendix  C. 


Table  12 

Suimnary  of  Preferences  Among 
Color -Risk  Ratings  Based  Upon  Median  Values 


A 

B 

Preference  Cortparison 

Blue -High 

Green - Moderate 

B  slightly  to  A 

Green -Low 

Blue -High 

A  moderately  to  B 

Blue -High 

Yellow- Low 

A  slightly  to  B 

Yellow-Moderate 

Green -High 

B  slightly  to  A 

Yellow -Low 

Green -High 

None 

Blue -Moderate 

Yellow- Low 

A  moderately  to  B 

Yellow- Low 

Green -Moderate 

B  slightly  to  A 

Green -Low 

Blue-ttoderate 

None 

Blue -High 

Yellow-Moderate 

A  slightly  to  B 

The  analysis  above  has  been  used  to  answer  the  five 
investigate  questions  first  posed  in  Chapter  I .  The  results 
will  be  used  to  draw  conclusions  in  Chapter  v.  The 
demographics  of  our  sanple  were  presented.  The  respondents 
were  categorized  into  risk  propensity  groups.  A  rank  order 
of  color- risk  ratings  was  established  for  the  entire  sample 
using  Friedman's  test.  Rank  orders  of  color-risk  ratings 
were  also  found  through  the  use  of  ordinal  scale 
comparisons.  Once  both  rankings  were  established, 

Spearman's  rank  correlations  were  used  to  compare  rank 
orders . 
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Next,  rank  orders  were  established  for  risk  propensity 
groups  using  the  Friedman  test.  The  rank  orders  were 
compared  using  the  Spearman  rank  correlations.  Rank  orders 
were  also  compared  across  item  summary  levels  using  the 
Friedman  test.  These  rank  orders  were  compared  using  the 
Spearman  ramk  correlation.  Finally,  the  pairwise  ranking 
preferences  were  compared  using  an  appropriate  ordinal 
scale. 

The  following  svimmary  highlights  significant  results  in 
the  analysis.  The  following  order  of  color-risk  ratings  was 
established:  Blue-Low,  Green -Low/Blue -Moderate,  Green- 
Moderate,  Blue-High,  Yellow-Low,  Green-High,  Yellow- 
Moderate,  and  Yellow-High.  No  difference  between  Green-Low 
and  Blue -Moderate  was  found  using  the  Friedman  test  or  the 
pairwise  comparison.  This  rank  order  structure  remained 
unchanged,  for  the  most  part,  for  all  risk  propensity 
categories  and  for  both  item  summary  levels.  The  Spearman's 
rank  correlations  showed  high  correlations  for  all 
combinations  of  r: sk  propensity  categories  and  for  both  item 
stimmary  levels.  Examination  of  median  values  indicated  that 
for  most  of  the  pairwise  rankings  one  choice  was  only 
slightly  preferred  to  the  other. 

Chapter  5  will  provide  conclusions  based  on  the  results 
presented  in  this  chapter.  Recommendations  for  topics  of 
further  study  will  also  be  discussed. 
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V.  Conclusions  and  Recoimnendations 


Introduction 

This  chapter  discusses  the  importance  of  the  research, 
reviews  the  methodology,  presents  the  major  findings, 
discusses  the  implications  and  limitations  of  the  findings, 
and  culminates  in  recommendations  for  source  selection  and 
future  study. 

Each  year  the  Air  Force  awards  weapon  system  contracts 
worth  billions  of  dollars  using  the  source  selection 
process.  In  the  source  selection,  performance  and  risk 
assessments  are  evaluated  to  award  a  contract  to  the 
proposal  that  gives  the  best  overall  value.  The  purpose  of 
this  research  was  to  investigate  the  performance/risk  trade¬ 
off  in  the  source  selection  decision  making  process.  The 
results  of  these  decisions  will  affect  the  military 
capability  and  readiness  of  the  Air  Force. 

Review  of  Methodology 

Air  Force  (AF)  acquisition  personnel  stationed  at 
Aeronautical  Systems  Center  (asc) ,  Wright -Patterson  AFB,  OH, 
who  had  participated  in  a  source  selection  were  surveyed. 
These  people  are  representative  group  of  all  AF  who  make 
decisions  during  a  source  selection.  The  survey  required 
respondents  to  rank  various  color- risk  rating  combinations 
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used  in  the  source  selection  process  from  most  to  least 
preferred.  The  rankings  were  used  to  estcd^lish  the  rank 
order  among  color-risk  ratings.  Respondents  also  indicated 
their  degree  of  preference  between  color- risk  rating  pairs 
using  an  ordinal  scale.  Use  of  the  ordinal  scale  provided 
another  means  to  establish  the  rank  order.  The  new  rank 
order  was  compared  to  the  first  rank  order  and  demonstrated 
the  reliability  of  the  survey.  Conparisons  were  made  for 
two  specific  areas  (item  stimmary  levels)  :  technical 
capability  and  management.  The  survey  contained  questions 
to  categorize  the  respondents  according  to  risk  propensities 
(risk  avoider,  risk  neutral  and  risk  taker)  to  measure  the 
effects  risk  propensity  has  on  the  color-risk  rank  order. 

Rank  orders  were  statisti ; ally  established  using  the 
Friedman  test  (a=0.05) .  A  rank  order  was  established  using 
the  results  of  the  median  value  for  pairwise  comparisons  of 
color-risk  ratings.  The  Spearman  correlation  test  was  used 
to  measure  the  similarities  in  the  various  rank  orders. 

Major  Findings 

A  definite  rank  order  exists  among  the  color-risk 
rating  combinations.  The  same  rank  order  was  found  using 
both  the  Friedman  test  and  the  ordinal  scale  method. 

Risk  propensity  does  not  change  the  relative  rank  order 
of  the  color-risk  ratings.  The  only  exception  is  that  risk 
takers  do  not  show  a  preference  between  the  Yellow- Low  and 


60 


Green-High  ratings,  while  risk  averters  and  risk  neutrals 
prefer  the  Yellow- Low  rating. 

The  item  being  evaluated  does  not  change  the  relative 
rank  order  of  the  color-risk  ratings.  The  rank  orders 
established  using  the  Friedman  test  for  the  technical 
capability  and  management  item  summaries  were  identical 
(Spearman  correlation  =  l.OOOO). 

The  ordinal  scale  pairwise  comparisons  showed  that  no 
preference  existed  between  Blue-Moderate/Green-Low  and 
Green-High/Yellow- Low.  A  moderate  preference  exists  for 
Blue-Moderate  to  Yellow-Low  and  Green-Low  to  Blue-High.  All 
other  combinations  examined  only  exhibited  slight 
preferences. 

Tcible  13  summarizes  the  rank  orders  found  for  the 
various  conditions  (item  summary  level  and  risk  propensity) 
and  methods  to  establish  rank  order  (Friedman  test  or 
ordinal  scale/median  value  method) .  Analysis  of  the  various 
rank  order  structures  indicated  that  a  definite  uniform  rank 
order  structure  exists.  Further  analysis  shows  that 
regardless  of  the  variables  studied  (risk  propensity  and 
item  being  evaluated) ,  the  rank  order  did  not  change. 

Implications  of  the  Findings 

The  findings  of  the  research  can  be  applied  to  the 
areas  of  personnel  assignment  for  source  selections  and 
decision  support  systems . 
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Table  13 


Summarization  of  Rank  Order  Findings 


Ordinal 
Scale 
Ranks : 
Tech  Cap 

Friedman 

Test 

Ranks : 
Tech  Cap 

Friedman 

Test 

Ranks: 

Mngmnt 

Friedman 

Test 

Ranks: 

Averters 

Friedman 

Test 

Ranks : 
Neutrals 

Friedman 

Test 

Ranks: 

Takers 

Blue- 

Low 

1 

1 

1 

1 

1 

1 

Blue- 

Moderate 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

Green- 

Low 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

2 (tie) 

Green- 

Moderate 

4 

4 

4 

4 

4 

4 

5 

5 

5 

5 

5 

5 

Yellow- 

Low 

6 (tie) 

6 

6 

6 

6 

6 (tie) 

7 

7 

7 

7 

6 (tie) 

Yellow- 

Moderate 

8 

8 

8 

8 

8 

8 

9 

9 

9 

9 

9 

9 

The  findings  of  the  research  show  that  individual  risk 
propensities  do  not  influence  the  performance- risk  trade-off 
decisions.  Some  program  directors  may  feel  risk  propensity 
is  an  important  factor  to  consider  when  selecting  people  for 
participation  in  a  source  selection  decision  making  role. 

The  research  indicates  that  individual  risk  propensity  does 
not  influence  the  decision  of  evaluators  when  choosing  among 
evaluated  proposals  based  on  color-risk  ratings.  When  faced 
with  making  decisions  based  on  color- risk  ratings,  one  can 
expect  that  the  decision  makers  will  lay  their  personal 
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attitudes  aside  and  arrive  at  a  decision  that  can  be 
predicted  with  the  rank  order  established  by  this  research. 

Since  a  uniform  rank  order  was  found,  it  could  be  used 
as  a  basis  for  a  decision  support  system  for  source 
selections.  In  generating  item  level  summaries,  a 
mathematical  weighting  system  could  be  employed  at  factor 
and  area  levels  to  arrive  at  an  aggregate  color- risk  rating. 
Using  this  system,  all  the  factors  requiring  analysis  could 
be  color- risk  rated  individually.  The  results  would  then  be 
input  into  the  decision  support  system  and  the  area,  item 
and  overall  color- risk  ratings  generated  according  to  a 
weighting  algorithm  contained  in  the  decision  support 
system.  Objective,  measurable  criterion,  not  intuition, 
would  determine  higher  level  color-risk  ratings.  Friedman 
tests  could  be  used  to  derive  the  interval  (weighting)  among 
color -risk  ratings  used  in  the  algorithm. 

Using  such  a  decision  support  system,  overall  color- 
risk  ratings  of  proposals  would  be  determined  from  lower 
level  assessments.  Comparing  color-risk  ratings  of  the 
evaluated  proposals  would  provide  comparative  scores  for 
proposals  that  could  be  used  in  choosing  a  winner  according 
to  the  color -risk  rating  rankings  found  by  this  research. 

This  approach  can  be  used  in  any  situation  where  a 
trade-off  between  two  scales  is  necessary  to  arrive  at  a 
decision.  Aircraft  maintainers  could  use  a  system  like  this 
to  determine  which  planes  to  fix  based  upon  logistics 
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constraints  (personnel,  parts,  time,  etc.)  and  mission 
readiness  requirements  (ie.  100%  of  squadron  aircraft 
flyable,  but  only  50%  mission  capedJle  versus  75%  of  squadron 
aircraft  flyable  and  mission  capable  with  the  remaining  25% 
down) .  The  inclement at ion  would  require  establishing  a  rank 
order  among  the  combinations  of  variables  and  determining 
the  appropriate  governing  algorithm. 

Limitations  of  the  Findings 

The  findings  of  this  thesis  are  limited  by  the  san^le 
surveyed.  The  survey  was  administered  to  personnel  at  ASC 
and  the  results  may  not  be  perfectly  reflective  of  all 
product  centers.  The  business  environments  at  various 
product  centers  is  different  because  of  the  type  of  weapon 
systems  procured.  ASC  buys  aircraft,  in  large  numbers, 
worth  billions  of  dollars,  where  Space  and  Missile  Center 
buys  satellites,  in  small  numbers,  worth  millions  of 
dollars.  The  size  of  the  purchase  invokes  regulatory 
constraints  as  well  as  managerial  opportunities  that  may  not 
be  present  elsewhere. 

The  findings  are  also  limited  to  source  selections 
involving  weapon  systems,  it  is  not  expandable  to  base  level 
and  spares  contracting.  Base  level  and  spares  contracting 
personnel  were  not  surveyed  to  develop  the  color- risk  rank 
order.  Base  level  contracting  typically  deals  with  very 
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small  dollar  amounts,  has  lower  relative  risk,  cmd  is 
subject  to  less  scrutiny  than  weapon  system  contracting. 

The  method  of  assessing  risk  propensity  also  limits  the 
findings.  The  respondents  were  asked  to  choose  between 
alternatives  in  a  lottery  type  setting.  The  dollar  values 
associated  with  these  choices  are  more  indicative  of 
personal  than  professional  business  situations.  This  could 
alter  the  view  a  program  director  has  of  a  person's  risk 
propensity  because  people  can  behave  differently  when 
dealing  with  risk  in  their  personal  and  professional  lives 
(15:176)  . 

Recommenda t ions 

This  thesis  was  the  first  attempt  to  examine  the 
preference  of  color -risk  ratings  in  the  source  selection 
environment.  Recommendations  from  this  research  consist  of 
applying  results  to  actual  source  selections,  expansion  on 
the  basic  framework  of  this  study,  and  investigation  of 
other  color- risk  rating  relationships. 

Recommendation  l:  The  decision  support  system 
presented  earlier  in  this  chapter  could  be  developed  and 
applied  to  an  actual  source  selection.  This  thesis 
conducted  an  ordinal  scale  study  of  the  color -risk  rating 
relationships  to  characterize  the  rank  structure,  interval 
level  data  for  a  decision  support  system  would  need  to  be 
developed  to  enhance  the  algorithm.  The  decision  support 
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system  would  also  have  to  build  the  interrelationships  among 
factors,  areas,  items  cuid  overall  ratings. 

Recononendation  2 :  This  research  examined  the 
relationship  of  technical  capability  ouid  management  item 
siiinmary  levels.  Further  research  should  examine  other  item 
summary  levels  (cost,  schedule,  contractual,  financial,  and 
past  performance)  auid  the  overall  color- risk  ratings  to 
compare  findings.  Broader  research  on  other  item  areas  will 
provide  a  better  understcinding  of  all  areas  considered  in 
the  source  selection. 

Recommendation  3 :  Air  Force  acquisition  personnel  were 
the  population  of  this  study,  with  Aeronautical  Systems 
Center  serving  as  the  target  sanqple.  Other  product  centers 
should  be  studied  to  examine  the  effects  on  source  selection 
decisions,  relating  to  color-risk  ratings,  to  account  for 
differences  in  technological  challenges  and  historical 
contracting  practices.  This  study  will  show  whether  or  not 
the  findings  of  this  study  are  generalizable.  It  also  has 
the  potential  to  identify  trends  in  risk  propensity  of 
product  centers. 

Recommendation  4:  An  unsolicited  comment  from  a  survey 
response  stated  that  the  acquisition  business  is  about 
taking  risks.  In  the  present  acquisition  environment,  it 
could  be  argued  that  the  goal  is  to  minimize  technical,  cost 
and  schedule  overruns  in  acquiring  weapon  systems.  It  could 
also  be  argued  that  the  goal  is  to  provide  the  most 
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technically  advanced  and  capable  weapon  systems.  These  two 
diametric  perspectives  on  acquisition  have  the  potential  to 
influence  source  selection  decisions.  An  examination  of 
individual  perceptions  on  the  goals  of  acquisition  can  yield 
further  insight  to  the  trade-offs  made  in  source  selections. 

Recoiranendation  5:  Noffsinger's  study  showed  that 
Technical  Risk  demonstrated  the  highest  correlation  with 
award  decisions  (18:79) .  His  studies  compared  all  winning 
proposals  with  all  losing  proposals.  The  results  of  this 
study  indicate  that  color  influences  performance -risk  trade¬ 
off  decisions.  Further  research  should  study  the  affects  of 
color  and  risk  ratings  on  actual  competitive  groups;  not  on 
an  aggregated  data  base,  as  Noff singer  did.  Study  of 
ratings  in  this  manner  will  reveal  how  performance  and  risk 
were  traded-off  between  competing  proposals.  For  exaxrple, 
Noff singer  points  out  that  three  Yellow- Low  proposals  were 
awarded  contracts  (18:57-58).  This  fact  raises  the  question 
of  "who  was  the  competition  and  how  did  their  color- risk 
ratings  affect  the  award  decision?" 

Recommendation  6:  This  study  included  all  possible 
color-risk  ratings  that  can  receive  an  award  (non-Red  color 
ratings) .  Noffsinger's  study  demonstrated  that  in  82  base 
level  source  selections  there  were  no  proposals  rated 
overall  as  Blue -Moderate,  Blue-High  and  Red-Low  (18:57-58) . 
The  question  arises  as  to  whether  these  color- risk  ratings 
have  real  meaning  in  the  source  selection  environment  or  are 
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they  just  rare  occurrences.  A  study  can  be  conducted  to 
investigate  this  phenomena  among  other  product  center  source 
selection  decisions.  The  results  have  the  potential  to 
modify  the  rank  order  preference  of  color- risk  ratings  found 
in  this  study  by  eliminating  "unrealistic"  ratings. 

Recommendation  7:  This  study  was  conducted  in  an 
environment  where  the  Department  of  Defense's  (DoD)  budget 
was  being  cut  and  Congressional  oversight  was  acute.  If  the 
acquisition  environment  conditions  change  (to  those  found  in 
the  Reagoui  Administration  for  example) ,  the  findings  of  this 
study  may  no  longer  be  applicable  and  a  re-examination  of 
this  topic  should  be  undertaken. 

Recommendation  8:  The  research  findings  show  that 
according  to  the  color -risk  rating  preferences,  source 
selection  participants  prefer  proposals  that  meet  or  exceed 
technical  requirements  at  low  risk.  A  question  that  lingers 
is  the  connotation  associated  with  low  risk  in  the  DoD 
acquisition  environment.  Does  low  risk  to  an  acquisition 
professional  mean  the  same  to  a  defense  contractor. 
Congressmen,  government  auditor,  or  the  average  American?  A 
study  should  be  undertaken  to  establish  if  the  connotated 
definition  of  low  risk  is  the  same  among  these  groups. 
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Appendix  A:  Survey 


Thank  you  for  participating  in  this  study.  We  realize  that  your  time  is 
extremely  valuable  and  greatly  appreciate  d>e  time  you  have  taken  to 
answer  this  survey.  This  survey  will  require  approximately  1 5  minutes  to 
complete.  The  purpose  of  this  research  effort  is  to  enhance  the  Source 
Selection  process.  The  results  will  provide  insight  on  the  role  Risk 
Management  plays  in  Source  Selection  decisions. 

You  have  been  selected  as  a  participant  because  of  your  involvement 
in  prior  Source  Selections.  Your  name  was  provided  by  our  Sponsor, 
Aeronautical  Systems  Center's  RFP  and  Source  Selection  Support  Program 
Office  (ASC/CYX).  Your  participation  is  completely  voluntary.  Please  do 
not  include  your  name  on  the  survey  so  anonymity  will  be  maintained. 

Responses  will  not  be  analyzed  on  an  individual  basis.  Individual 
responses  will  be  grouped  so  survey  results  will  be  a  reflection  of  the 
sample  population.  Findings  will  be  published  this  September  in  AFIT  thesis, 
AFIT/GSM/LAS/93S-18. 

Again,  thank  you  for  your  participation. 


Section  I 

1 .  Which  of  the  following  would  you  prefer? 

A.  An  80%  chance  of  gaining  $4,000  and  a  20%  chance  of  gaining  $0 

B.  A  certain  gain  of  $3,200 

C.  No  preference 

2.  Which  of  the  following  would  you  prefer? 

A.  A  25%  chance  of  gaining  $3,200  and  a  75%  chance  of  gaining  $0 

B.  A  20%  chance  of  gaining  $4,000  and  an  80%  chance  of  gaining  $0 

C.  No  preference 
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3.  Which  of  the  following  would  you  prefer? 

A.  An  80%  chance  of  losing  $4,000  and  a  20%  chance  of  losing  $0 

B.  A  certain  loss  of  $3,200 

C.  No  preference 

4.  Which  of  the  following  would  you  prefer? 

A.  A  25%  chance  of  losing  $3,200  and  a  75%  chance  of  losing  $0 

B.  A  20%  chance  of  losing  $4,000  and  an  80%  chance  of  losing  $0 

C.  No  preference 

5.  Which  of  the  following  would  you  prefer? 

A.  A  90%  chance  of  gaining  $3,000  and  a  10%  chance  of  gaining  $0 

B.  A  45%  chance  of  gaining  $6,000  and  an  55%  chance  of  gaining  $0 

C.  No  preference 

6.  Which  of  the  following  would  you  prefer? 

A.  A  90%  chance  of  losing  $3,000  and  a  10%  chance  of  losing  $0 

B.  A  45%  chance  of  losing  $6,000  and  an  55%  chance  of  losing  $0 


C.  No  preference 


Section  II 

The  following  are  the  current  AFR  70-1 5,  Formal  Source  Selections  for 
Major  Acquisitions,  and  AFR  70-30,  Streamlined  Source  Selection 
Procedures,  definitions.  They  represent  the  official  definitions  of  the  color 
code  ratings  and  proposal  risk  assessments.  Please  use  these  definitions  for 
the  next  two  sections. 


COLOR  RATINGS 

BLUE  Exceeds  specified  performance  or  capability  in  a 

(Exceptional)  beneficial  way  to  the  Air  Force;  and  has  a  high 

probability  of  satisfying  the  requirement;  and  has  no 
significant  weakness. 


GREEN 

(Acceptable) 


Meets  evaluation  standards;  and  has  good  probability 
of  satisfying  the  requirement;  and  any  weaknesses  can 
be  readily  corrected. 


YELLOW 

(Marginal) 


Fails  to  meet  evaluation  standards;  or  has  low 
probability  of  satisfying  the  requirement;  or  has 
significant  deficiencies  but  correctable. 


RED 

(Unacceptable) 


Fails  to  meet  a  minimum  requirement 
(Not  used  in  this  survey.) 


PROPOSAL  RISK  ASSESSMENTS 

HIGH  Likely  to  cause  significant  serious  disruption  of 

schedule,  increase  in  cost,  or  degradation  of 
performance  even  with  special  contractor  emphasis 
and  close  government  monitoring. 


MODERATE  Can  potentially  cause  some  disruption  of  schedule, 
increase  in  cost,  or  degradation  of  performance. 
However  special  contractor  emphasis  or  close 
government  monitoring  will  probably  be  able  to 
overcome  difficulties. 

LOW  Has  little  potential  to  cause  disruption  of  schedule, 

increase  in  cost,  or  degradation  of  performance. 
Normal  contractor  effort  and  normal  government 
monitoring  will  probably  be  able  to  overcome 
difficulties. 
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The  following  table  is  the  Source  Selection  Evaluation  Matrix.  It  is  a  matrix 
that  contains  possible  combinations  of  color  rankings/proposat  risk 
assessment  pairs.  Please  rank  the  boxes  according  to  your  preferences  in 
making  a  source  selection  decision.  Number  1  and  Number  9  have  been 
entered  to  illustrate  the  procedure  for  completing  this  question.  Feel  free  to 
change  these  rankings  if  you  want,  in  ranking  the  pairs.  Number  1  is  your 
most  preferred  option  while  number  9  is  your  least  preferred  option.  Ties 
are  not  allowed. 


Please  evaluate  for  Item  Summary: 
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Please  evaluate  for  Item  Summary: 


Management 


COLOR 

RATING 

RISK 

BLUE 

GREEN 

YELLOW 

HIGH 

9 

MODERATE 

LOW 

1 

Section  III  -  in  this  section,  imagine  yourself  as  the  Source  Selection 
Authority  and  each  color  rating/proposal  risk  assessment  represents  the 
technical  capability  item  area  ranking  of  competing  proposals.  Select  the 
proposal  you  prefer  and  the  degree  to  which  you  prefer  it  over  the  other 
proposal.  Please  circle  a  number  on  the  scale. 

A. 

Color:  Blue  _ 

Risk:  High  1  2 

Highly 
Prefer 
Thie  Choice 


_  Color:  Blue 

7  Risk:  High 

Highly 
Prefer 
This  Choice 


_ Color:  Yellow 

1  2  3  4  5  6  7  Risk:  Low 

Highly  No  Highly 

Prefer  Preference  Prefer 

This  Choice  This  Choice 


_  Color:  Green 

7  Risk:  Moderate 

Highly 

Prefer 

This  Choice 


E. 

Color:  Yellow  _ 

Risk:  Moderate  1  2 

Highly 

Prefer 

This  Choice 


_  Color:  Green 

7  Risk:  High 

Highly 

Prefer 

This  Choice 


_  Color:  Yellow 

7  Risk:  Low 

Highly 
Prefer 
This  Choice 


F. 

Color:  Yellow 

Risk:  Low 

1  2 

3 

4 

Highly 

No 

Prefer 

Preference 

This  Choice 

G. 

Color:  Blue 

Risk:  Moderate  1  2 

3 

4 

Highly 

No 

Prefer 

Preference 

This  Choice 

Color:  Green 

4  5 

6 

7  Risk:  High 

No 

Highly 

Preference 

Prefer 

This  Choice 

D. 

Color:  Blue  _ 

Risk:  Low  1  2  3  4  5  6 

Highly  No 

Prefer  Preference 

This  Choice 


c. 

Color:  Blue 
Risk:  High 


B. 

Color:  Green 

Risk:  Low 

1  2 

3 

4 

Highly 

No 

Prefer 

Preference 

Thie  Choice 

Color:  Green 

4  5 

6 

7  Risk:  Moderate 

No 

Highly 

Preference 

Prefer 

This  Choice 

H. 

Color:  Yellow 

Risk:  Low 

1  2 

3 

4 

Highly 

No 

Prefer 

Preference 

This  Choice 

_  Color:  Green 

7  Risk:  Moderate 

Highly 

Prefer 

This  Choice 


1  e 

Color:  Yellow 

Risk:  Moderate  1  2 

3 

4 

Highly 

No 

Prefer 

Preference 

This  Choice 

_  Color:  Green 

7  Risk:  Low 

Highly 
Prefer 
This  Choice 


J. 

Color:  Green 
Risk:  Low 


Color:  Blue 

1  2 
Highly 

Prefer 

This  Choice 

3 

4  5 

No 

Preference 

6 

7  Risk:  Moderate 

Highly 

Prefer 

This  Choice 

K. 

Color:  Blue 
Risk:  High 


Color:  Yellow 

1  2 
Highly 

Prefer 

This  Choice 

3 

4  5 

No 

Preference 

6 

7  Risk:  Moderate 

Highly 

Prefer 

This  Choice 

Section  IV 

1 .  What  is  your  military  rank/civiiian  grade?  _ 

2.  How  many  total  years  of  acquisition  experience  do  you  have? 

3.  Which  acquisition  specialty  are  you  presently  working  in?  (ie.  program 
management,  logistics,  contracting,  program  control,  etc.) 

4.  In  your  personal  financial  life,  do  you  consider  yourself: 

A.  risk  taking  B.  risk  neutral  C.  risk  avoiding 

5.  in  your  professional  life,  do  you  consider  yourself: 

« 

A.  risk  taking  B.  risk  neutral  C.  risk  avoiding 
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6.  What  is  the  highest  position  you  have  held  in  a  source  selection? 


A.  SSA 

B.  SSEB/SSET  Chairperson 

C.  Item  Captain/Area  Chief 

D.  Evaluator 

E.  Other,  please  specify:  _ 

Please  place  in  return  envelope.  Thank  you  for  participating! 
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DEPARTMENT  OF  THE  AIR  FORCE 

HEAOaUAflTEM  AERONAUTICAL  SYSTEMS  CENTER  (AFMC) 
WRIQHT-RATTERSON  AW  FORCE  BASE.  OHIO 


FROM;  ASC/CY  *1 3  1883. 

SUBJ;  Survey  of  Source  Selection  Participants 
TO:  Survey  Participants 

1.  I  support  this  AFIT  thesis  project  which  is  sponsored 
by  the  RFP  and  Source  Selection  Support  Program  Office 
(ASC/CYX) . 

2.  The  attached  survey  evaluates  AF  Regulations 

70-15  and  70-30  emphasis  on  the  inportance  of  identifying 
and  evaluating  risk  in  the  source  selection  process.  As 
a  recipient  of  this  survey,  you  are  in  a  position  to 
contribute  to  a  research  project  investigating  how  risk  is 
dealt  with  by  source  selection  participants.  The  data 
collected  will  be  used  to  help  understand  how  one's  attitude 
toward  risk  influences  the  decisions  of  source  selection 
participants. 

3.  Please  take  10-15  minutes  to  conplete  the  attached 
survey  and  return  it  in  the  enclosed  addressed  envelope  no 
later  than  24  May  93.  We  request  that  you  do  not  write  your 
name  on  the  survey,  so  your  individual  response  will  remain 
anonymous . 

4.  Your  participation  in  this  study  is  conpletely 
voluntary,  but  we  would  greatly  appreciate  your  help.  For 
further  information,  please  contact  faculty  advisors  for 
this  research  project,  Lt  Col  Carl  Tenplin,  54845,  or 

Maj  Kevin  Grant,  54845. 

ROBERT  C.  KELT  2  Atch 

Colonel ,  USAF  1 .  Survey 

Director,  Program  Management  2.  Return  Envelope 
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Appendix  g; _ Friedman  Test  Results 


Analysis  for  entire  sample.  ORIGIN  ^  1 

M  -  READPRNI  SUMMARY  ) 

cols(  M  )  =  18  rows!  M  )  =  76 

1=1..  rows!  M  )  J  =  1  ..  cols(  M  ) 

At  this  stage,  we  are  calculating  the  sum  of  the  ranks  for  each  color-risk  rating. 

This  is  accomplished  by  summing  the  columns  of  the  matrix.  SMBL  stands 
for  the  sum  of  management  blue  low  column.  STBL  stands  for  the  sum  of 
technical  blue  low  column.  (G  =  green,  Y= yellow,  M  =  moderate,  H  =  high) 


SMBL 

=  E“,.. 

1 

SMBL  =  76 

STBL 

"  ^  10 

1 

STBL  =  76 

SMBM 

1 

SMBM  =  207 

STBM 

=  Em.,11 

1 

STBM  =  199 

SMBH 

1 

SMBH  394 

STBH 

3~!  ^1. 12 

1 

STBH  =  395 

SMGL 

-■  S 

1 

SMGL  =  201 

STGL 

^1. 13 

1 

STGL  =  204 

SMGM 

1 

SMGM  »  334 

STGM 

1 

STGM  =  323 

SMGH 

S  M..6 

1 

SMGH  =516 

STGH 

1 

STGH  =512 

SMYL 

1 

SMYL  =  447 

STYL 

y!  ^1. 16 

1 

STYL  =  460 

SMYM 

= 

1 

SMYM  =  561 

STYM 

1 

STYM  =  567 

SMYH 

=  IH., 

■ 

SMYH  =  684 

STYH 

‘  y~!  ^1.18 

STYH  =  682 

I  I 


Analysis  step  to  simplify  calculations. 


s, 

SMBL 

S5 

SMGM 

S10 

STBL 

s,* 

STGM 

s. 

SMBM 

Se  ^ 

SMGH 

s„ 

=  STBM 

S,e  ^ 

STGH 

S3 

=  SMBH 

S7 

SMYL 

S,3 

STBH 

S,6  ^ 

STYL 

S4 

=  SMGL 

Ss 

SMYM 

S,3 

-  STGL 

S,7  ^ 

STYM 

s. 

SMYH 

S,8 

STYH 
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Calculation  of  the  test  statistics. 


Sum  of  the  squares  for  the  matrix  elements.  This  term  is  the  same  for  both 
management  and  technical  ratings.  The  term  cols(M)/2  is  used  because  the  file  read  in 
contains  two  sets  of  data:  management  and  technical  rankings. 


A2 


rows!  M ) 


cols(  M ) 


colsl  M  )  i  <  o  cols(  M  ) 

2  '  \  •  2-  2  ’ 


6 


A2  =  2.166-10* 


Square  of  the  column  sums  added  together  divided  by  the  number  of  cases.  B2M  is  for  the 
management  case.  B2T  is  for  the  technical  case. 


K  =  1  ..  9 


10..  18 


1 


2  ' 


B2M  -  - A  Sk  : 


rows!  M 


B2T 


K 


1 

rows!  M  ) 


Us 


B2M  =  2.111-10* 


B2T  =  2.115-10* 


Calculation  of  the  F-statistic  for  the  null  hypothesis.  (Null  hypothesis  is  that  there  is  no  difference 
among  the  different  color-risk  ratings)  The  confidence  level  required  is  95%.  T2M  is  the 
F-statistic  for  management  color  risk  rating  and  T2T  is  the  F-statistic  for  technical  color-risk  rating. 


(  rows!  M  ) 

T2M 


,  .  colsl  M  )  colsl  M  ) 

rowsi  M  )•  • 

B2M  - - j  - - -  ^ 

4 

A2  'MM 


T2M  =  548.309 


I  rows!  M  ) 

T2T 


colsl  M  )  colsl  M  ) 

rowsi  M  )• 

).i  B2T - ^  ±  - 

L  ^ 

A2  B2t 


T2T  =  600.889 


Calculation  of  a  95%  confidence  level  statistic. 


k1 

colsl  M  )  2 

2 

00 

II 

k2 

kM  rows!  M  )  1  ) 

k2  =600 
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Using  Statistix  4.0  and  the  values  of  kl  and  k2  above  for  the  degrees  of  freedom,  the  F-statistic 
value  is  1 .95.  Since  T2M  and  T2T  are  greater  than  the  F-statistic  for  a  95%  confidonce  interval,  the 
null  hypothesis  is  rejected.  The  comparison  statistic  is  calculated  to  identify  significant  differences 
in  rankings. 

Comparison  statistic.  LevelM  is  the  comparison  statistic  for  the  management  color-risk  rankings 
and  LevelT  is  for  the  technical  color-risk  rankings. 


1  ' 


LevelM  ==  22.99 


2 
•  2 

0  : 


LevelT  =  22.078 


LevelT 


2  rows(  M  )•(  A2  -  B2T  ) 

/  i  »  /  cols(  M  ) 

(  rows(  M  )  -  1  )•  I - ^ - 
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Analysis  for  risk  avoiders. 


ORIGIN  =  1 


M  ^  READPRNI  AVOID  ) 

colsl  M  )  =  18  rows!  M  )  =  23 

1  =  1..  rows!  M  )  J  =  1  ..  cols(  M  ) 

At  this  stage,  we  are  calculating  the  sum  of  the  ranks  for  each  color-risk  rating. 
This  is  accomplished  by  summing  the  columns  of  the  matrix.  SM6L  stands 
for  the  sum  of  management  blue  low  column.  STBL  stands  for  the  sum  of 
technical  blue  low  column.  (G  =  green,  Y = yellow,  M  =  moderate,  H  =  high) 


SMBL 

= 

1 

SMBL 

*  23 

STBL 

II 

-M 

j: 

o 

STBL  =  23 

SMBM 

1 

SMBM 

=  58 

STBM 

=  1]“'.” 

1 

STBM  =  56 

SMBH 

"  y!  ^1.3 

1 

SMBH 

=  118 

STBH 

y  ^1. 12 

1 

STBH  =  119 

SMGL 

1 

SMGL 

=  65 

STGL 

y  ^1. 13 

1 

STGL  =  64 

SMGM 

1 

SMGM 

=  101 

STGM 

1 

STGM  =  98 

SMGH 

Em,  6 

1 

SMGH  : 

=  162 

STGH 

E'^'-’b 

1 

STGH  =  159 

SMYL 

= 

1 

SMYL  = 

=  133 

STYL 

y  ^1. 16 

1 

STYL  =  137 

SMYM 

1 

SMYM 

=  168 

STYM 

1 

STYM  =  172 

SMYH 

=  2“.,. 

SMYH  = 

=  207 

STYH 

■  E  ^'-’8 

STYH  =  207 

Analysis  step  to  simplify  calculations. 


s, 

=  SMBL 

Sg  SMGM 

Sio 

STBL 

S,4  - 

STGM 

Sj 

=  SMBM 

Sg  =  SMGH 

s„ 

=  STBM 

S16  ' 

STGH 

S3 

SMBH 

S7  SMYL 

S,2 

STBH 

s,«  - 

STYL 

S4 

=  SMGL 

Sg  -  SMYM 

Sl3 

=  STGL 

S,7  ^ 

STYM 

s. 

SMYH 

S,B 

=  STYH 
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Calculation  of  the  test  statistics. 


Sum  of  the  squares  for  the  matrix  elements.  This  term  is  the  same  for  both 
management  and  technical  ratings.  The  term  cois(M)/2  is  used  because  the  file  read  in 
contains  two  sets  of  data;  management  and  technical  rankings. 


A2 


'  .  1  .  2  '=°!*'“’  .  1 


A2  •  6.555-10’ 

Square  of  the  column  sums  added  together  divided  by  the  number  of  cases.  B2M  is  for  the 
management  case.  B2T  is  for  the  technical  case. 

K  =  1  ..  9  L  10..  18 


B2M 


1 


rows!  M ) 
B2M  =6.402-10^ 


K 


B2T 


1 _ 

rows!  M  ) 


B2T  =  6.435*10^ 


Calculation  of  the  F-statistic  for  the  null  hypothesis.  (Null  hypothesis  is  that  there  is  no  difference 
among  the  different  color-risk  ratings)  The  confidence  level  required  is  95%.  T2M  is  the 
F-statistic  for  management  color  risk  rating  and  T2T  is  the  F-statistic  for  technical  color-risk  rating. 


T2M 


I  rows(  M ) - A---  •;  y  ' 

(rows(M)  1  )  |  B2M  -  - f_  - 

_ _ ^ _ _  _  _ _ 4 

A2  B2M 


T2M  =  176.601 


T2T 


....  cols(  M  )  ,  cols(  M  )  . 

rows(  M  )•  -  '  ?  ^ —  f  1 

(  rowsi  M  )  1  )  •  B2T  - - - — - - - - - - 

i  4 


T2T  =  231.367 


Calculation  of  a  95%  confidence  level  statistic. 


k1 

cols(M)  2 

2 

k1  8 

k2 

-  k1(  rows(  M  )  1  ) 

k2  =176 
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Using  Statistix  4.0  and  the  values  of  k1  and  k2  above  for  the  degrees  of  freedom,  the  F-statistic 
value  is  1.95.  Since  T2M  and  T2T  are  greater  than  the  F-statistic  for  a  95%  confidence  interval,  the 
null  hypothesis  is  rejected.  The  comparison  statistic  is  calculated  to  identify  significant  differences 
in  rankings. 


Comparison  statistic.  LevelM  is  the  comparison  statistic  for  the  management  color-risk  rankings 
and  LevelT  is  for  the  technical  color-risk  rankings. 


LevelM 


1.95-i 


2  rows(  M  )•(  A2  B2M 


(  rows  I  M  )  -  1  )  • 


colsl  M ) 


1 


LevelM  =  12.326 


LevelT 


1.95- 


2  rows(  M  )•(  A2  B2T  ) 


( rows!  M  )  - 


.  .  I  colsl  M  1 
11—2— 


1 

,  2 


LevelT  =  10.913 


* 
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Analysis  for  risk  neutrals. 


ORIGIN  =  1 


M  READPRNI NEUTRAL ) 

cols(  M  )  *  18  rows!  M  )  =  29 

I  -  1  ..  rows!  M  )  J  -  1..  colsIM) 

At  this  stage,  we  are  calculating  the  sum  of  the  ranks  for  each  color-risk  rating. 

This  is  accomplished  by  summing  the  columns  of  the  matrix.  SMBL  stands 
for  the  sum  of  management  blue  low  column.  STBL  stands  for  the  sum  of 
technical  blue  low  column.  (G  =  green,  Y  =  yellow,  M  =  moderate,  H  =  high) 


SMBL 

=  x;“" 

1 

SMBL  =  29 

stbl  ^  J;m,  ,0 

1 

STBL  =  29 

SMBM 

=  1]“' » 

1 

SMBM  =  80 

STBM  =  ^M,  „ 

1 

STBM  =  78 

SMBH 

1 

SMBH  =  152 

stbh  =  „ 

1 

STBH  =  155 

SMGL 

=  * 

1 

SMGL  =  73 

stgl  =  ,3 

•  1 

STGL  =  74 

SMGM 

1 

SMGM  s  129 

STGM  =  5]m, 

1 

STGM  =  1 26 

SMGH 

=  S'*''." 

1 

SMGH  =  196 

STGH  ^  5]m,.,b 

1 

STGH  =  198 

SMYL 

1 

SMYL  =  170 

STYL  = 

1 

STYL  =  171 

SMYM 

=  !]“'• 

1 

SMYM  =215 

STYM  ^  J]m,.„ 

1 

STYM  =213 

SMYH 

=  • 

1 

SMYH  =  261 

STYH  =  5]M,.,3 

1 

STYH  =  261 

Analysis  step  to  simplify  calculations. 

s, 

SMBL  Sb 

SMGM 

S,o 

STBL  S,4  -  STGM 

Sj 

=  SMBM  Sq 

^  SMGH 

s„ 

-  STBM  S,5  =  STGH 

S3 

^  SMBH  S7 

SMYL 

S12 

STBH  S,o  STYL 

S4 

^  SMGL  Sg 

^  SMYM 

Sl3 

STGL  S„  -  STYM 

Sg  SMYH 

S18  -  STYH 
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Calculation  of  the  test  statistics. 


Sum  of  the  squares  for  the  matrix  elements.  This  term  is  the  same  for  both 
management  and  technical  ratings.  The  term  cols(M)/2  is  used  because  the  file  read  in 
contains  two  sets  of  data;  management  and  technical  rankings. 


rows(M)  ?^^^-(‘=^'^»  ^  1  .  1 

#  A2  - = - ^ - - - - - - 


A2  =  8.265-10^ 


Square  of  the  column  sums  added  together  divided  by  the  number  of  cases.  B2M  is  for  the 
management  case.  B2T  is  for  the  technical  case. 


K  ^  1  ..  9 


B2M 


_1 

rows(  M  ) 


2  i 


[  K 


L  - 
B2T 


10..  18 
1 

rows!  M  ) 


B2M  =8.068-10=* 


B2T  =  8.077-10=* 


Calculation  of  the  F-statistic  for  the  null  hypothesis.  (Null  hypothesis  is  that  there  is  no  difference 
among  the  different  color-risk  ratings)  The  confidence  level  required  is  95%.  T2M  is  the 
F-statistic  for  management  color  risk  rating  and  T2T  is  the  F-statistic  for  technical  color-risk  rating. 


<  rows(  M  )  -  1  )•<  B2M 


T2M 


rows(  M  ) 
A2  B2M 


colsl  M  )  cols(  M  ) 
^  ■  2 

. 4“ 


T2M  =  219.526 


(  rows(  M  )  1  ) 


T2T 


rows!  M  ) 


colsl  M  )  colsl  M  ) 


B2T 


A2  B2T 


T2T  =  231.339 

Calculation  of  a  95%  confidence  level  statistic. 


k1 

colsl  M  )  2 

2 

k1  =  8 

k2 

*  k1  -1  rows!  M  )  1  ) 

k2  =  224 
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Using  Statistix  4.0  and  the  values  of  k1  and  k2  above  for  the  degrees  of  freedom,  the  F-statistic 
value  is  1 .95.  Since  T2M  and  T2T  are  greater  than  the  F-statistic  for  a  95%  confidence  interval,  the 
null  hypothesis  is  rejected.  The  comparison  statistic  is  calculated  to  identify  significant  differences 
in  rankings. 


Comparison  statistic.  LevelM  is  the  comparison  statistic  for  the  management  color-risk  rankings 
and  LevelT  is  for  the  technical  color-risk  rankings. 


LevelM 


1.95 


2  rows(  M  )-(  A2  -  B2M  } 


{  rows!  M  )  -  1  )  ■ 


cols(  M  ) 


% 


LevelM  =  13.921 


LevelT 


1.95- 


2  rows(  M  )•(  A2  -  B2T  » 


(  rows!  M  )  -  1  )• 


colsl  M  1 


I 

l2 


LevelT  =  13.6 


i 


4 
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Analysis  for  risk  takers. 


ORIGIN  =  1 


M  -  REAOPRNI  RISKY  ) 

cols(  M  )  =18  rows(  M  )  =  24 

I  I..  rows!  M  )  J  =  1  ..  colsl  M  ) 

At  this  stage,  we  are  calculating  the  sum  of  the  ranks  for  each  color-risk  rating. 
This  is  accomplished  by  summing  the  columns  of  the  matrix.  SMBL  stands 
for  the  sum  of  management  blue  low  column.  STBL  stands  for  the  sum  of 
technical  blue  low  column.  <G  =  green,  Y  =  yellow,  M  =  moderate,  H  =  high) 


SMBL 

=  S“'.' 

1 

SMBL  =  24 

STBL  =  ^M,  ,0 

1 

STBL  =  24 

SMBM 

1 

SMBM  =  69 

STBM  =  11 

1 

STBM  =  65 

SMBH 

-  Em., 3 

1 

SMBH  =  124 

STBH  ^  5]m,  ,2 

1 

STBH  =  121 

SMGL 

^  E“' « 

1 

SMGL  =63 

STGL  = 

1 

STGL  =  66 

SMGM 

=  E“'  = 

1 

SMGM  =  104 

STGM  --  5]M, 

1 

STGM  =  99 

SMGH 

E”'.= 

1 

SMGH  = 158 

STGH  =  5]m,,i, 

1 

STGH  =  155 

SMYL 

-  E“'.’ 

1 

SMYL  =  144 

styl  „ 

1 

STYL  =  152 

SMYM 

^  E“..« 

1 

SMYM  =  178 

STYM  ^  ^M,  ,7 

1 

STYM  =  182 

SMYH 

=  E“'.« 

1 

SMYH  =216 

STYH  =  2m,i3 

1 

STYH  =  214 

Analysis  step  to  simplify  calculations. 

s, 

SMBL  Sg 

SMGM 

S,o 

^  STBL  S,4  STGM 

S2 

-  SMBM  Se 

-  SMGH 

s„ 

-  STBM  S,3  -  STGH 

S3 

SMBH  S7 

SMYL 

S,2 

-  STBH  S,3  STYL 

S4 

SMGL  Sg 

SMYM 

S,3 

-  STGL  S,7  -  STYM 

S9  SMYH 

S,8  STYH 

Calculation  of  the  test  statistics. 


Sum  of  the  squares  for  the  matrix  elements.  This  term  is  the  same  for  both 
management  and  technical  ratings.  The  term  cols(M)/2  is  used  because  the  file  read  in 
contains  two  sets  of  data;  management  and  technical  rankings. 


A2 


rows!  M  )■ 


cols(  M  ) 
2 


/  cols(  M  ) 
V  2 

6 


1 

>  \  2 


I'l 


« 


% 


A2  =6.84*10® 


Square  of  the  column  sums  added  together  divided 
management  case.  B2T  is  for  the  technical  case. 

K  =  1  ..  9  L  - 


B2M 


1 

rows(  M  ) 


2  i 


B2T 


B2M  =6.647*10® 


B2T 


by  the  number  of  cases. 


10..  18 
1 

rows(  M  ) 


=  6.652*10® 


B2M  is  for  the 


I 

I 

J 


Calculation  of  the  F-statistic  for  the  null  hypothesis.  (Null  hypothesis  is  that  there  is  no  difference 
among  the  different  color-risk  ratings)  The  confidence  level  required  is  95%.  T2M  is  the 
F-statistic  for  management  color  risk  rating  and  T2T  is  *he  F-statistic  for  technical  color-risk  rating. 


T2M 


(  rows!  M)  -  1)1  B2M 

_ L 


, .. ,  colsl  M  )  /  cols(  M  ) 
rows(  M  ) - ^ - [ - 2 - 

4 

A2  B2M 


1 


2  1 

i 


I 

I 


T2M  =  148.977 


( rows(  M  )  -  1  )• 


T2T 


B2T 


, . . ,  colsl  M  )  /  colsl  M  )  , 

rowsIM) - 2 — ^  ^  1 


A2  B2T 


T2T  =  153.17 


Calculation  of  a  95%  confidence  level  statistic. 


k1 

colsl  M  )  2 

2 

k1  =  8 

k2 

-  k1  -1  rows!  M  )  -  1  ) 

k2  =184 
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Using  Statisttx  4.0  and  the  values  of  k1  and  k2  above  for  the  degrees  of  freedom,  the  F-statistic 
value  is  1 .95.  Since  T2M  and  T2T  are  greater  than  the  F-statistic  for  a  95%  confidence  interval,  the 
null  hypothesis  is  rejected.  The  comparison  statistic  is  calculated  to  identify  significant  differences 
in  rankings. 


» 

Comparison  statistic.  LevelM  is  the  comparison  statistic  for  the  management  color-risk  rankings 
and  LevelT  is  for  the  technical  color-risk  rankings. 

« 


LevelM 


1.95- 


2  rows(  M  )•(  A2  -  B2M  ) 


(  rows!  M  )  -  1  )■ 


colsl  M  ) 


1 

2 


LevelM  =  13.822 


LevelT 


1.95- 


2  rows(  M  )•(  A2  -  B2T) 


(  rows!  M  )  -  1  )• 


colsl  M  ) 


-  1 


1 


1^ 

2 


LevelT  =  13.656 
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Response  Distribution  by  Risk  Types 

Question  A  (Blue  High-Green  Moderate) 


BH  GM 


E 

1  Risk  Avoiders  Risk  Neutrals  >$$$ 

1  Risk  Takers 
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Frequency 


Response  Distribution  by  Risk  Types 

Question  B  (Green  Low-Blue  High) 
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Response  Distribution  by  Risk  Types 

Question  C  (Blue  High-Yellow  Low) 


1  2  3  4  5  6  7 


BH  YL 

Risk  Avoiders  V/ZA  Risk  Neutrals  Risk  Takers 


Response  Distribution  by  Risk  Types 

Question  E  (Yellow  Moderate^reen  High) 
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Response  Distribution  by  Risk  Types 

Question  F  (Yellow  Low-Green  High) 


Risk  Avoiders  Y//A  Risk  Neutrals  Risk  Takers 


■» 


% 
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Response  Distribution  by  Risk  Types 

Question  G  (Blue  Moderate-Yellow  Low) 


BM  YL 


r 


Risk  Avoiders  Y//\  Risk  Neutrals  Risk  Takers 


# 
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Response  Distribution  by  Risk  Types 

Question  H  (Yellow  Low-Green  Moderate) 


Frequency 


Response  Distribution  by  Risk  Types 

Question  J  (Green  Low-Blue  Moderate) 
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Response  Distribution  by  Risk  Types 

Question  K  (Blue  High-Yellow  Moderate) 
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